{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "876db82e",
   "metadata": {},
   "source": [
    "# Prot T5 Finetuning\n",
    "# per residue classification\n",
    "\n",
    "This notebook allows you to finetune ProtT5 to your own datasets\n",
    "\n",
    "The protein language model ProtT5 was first published [here](https://ieeexplore.ieee.org/document/9477085) and is available on [github](https://github.com/agemagician/ProtTrans). We use the [huggingface](https://huggingface.co/Rostlab/prot_t5_xl_uniref50) checkpoint.\n",
    "\n",
    "For better perfomance we apply [Parameter-Efficient Fine-Tuning (PEFT)](https://huggingface.co/blog/peft). For this we apply [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685).\n",
    "\n",
    "For higher memory efficiency we also utilize the [deepspeed](https://github.com/microsoft/DeepSpeed) implementation of [huggingface](https://huggingface.co/docs/accelerate/usage_guides/deepspeed).\n",
    "\n",
    "The core training loop is implemented with the pytorch [huggingface trainer](https://huggingface.co/docs/transformers/main_classes/trainer) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a5ec35",
   "metadata": {},
   "source": [
    "## Imports and env. variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "angry-toronto",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import dependencies\n",
    "import os.path\n",
    "os.chdir(\"set a path here\")\n",
    "\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import copy\n",
    "\n",
    "import transformers, datasets\n",
    "from transformers.modeling_outputs import TokenClassifierOutput\n",
    "from transformers.models.t5.modeling_t5 import T5Config, T5PreTrainedModel, T5Stack\n",
    "from transformers.utils.model_parallel_utils import assert_device_map, get_device_map\n",
    "from transformers import T5EncoderModel, T5Tokenizer\n",
    "from transformers import TrainingArguments, Trainer, set_seed\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "\n",
    "from evaluate import load\n",
    "from datasets import Dataset\n",
    "\n",
    "from tqdm import tqdm\n",
    "import random\n",
    "\n",
    "from scipy import stats\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8534fbd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set environment variables to run Deepspeed from a notebook\n",
    "os.environ[\"MASTER_ADDR\"] = \"localhost\"\n",
    "os.environ[\"MASTER_PORT\"] = \"9993\"  # modify if RuntimeError: Address already in use\n",
    "os.environ[\"RANK\"] = \"0\"\n",
    "os.environ[\"LOCAL_RANK\"] = \"0\"\n",
    "os.environ[\"WORLD_SIZE\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "808bb08d",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Environment to run this notebook\n",
    "\n",
    "\n",
    "These are the versions of the core packages we use to run this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b35bdadd",
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Torch version:  1.13.1\n",
      "Cuda version:  11.7\n",
      "Numpy version:  1.22.3\n",
      "Pandas version:  1.5.3\n",
      "Transformers version:  4.26.1\n",
      "Datasets version:  2.9.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Torch version: \",torch.__version__)\n",
    "print(\"Cuda version: \",torch.version.cuda)\n",
    "print(\"Numpy version: \",np.__version__)\n",
    "print(\"Pandas version: \",pd.__version__)\n",
    "print(\"Transformers version: \",transformers.__version__)\n",
    "print(\"Datasets version: \",datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d109a4d7",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**For easy setup of this environment you can use the finetuning.yml File provided in this folder**\n",
    "\n",
    "check here for [setting up env from a yml File](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2dda19",
   "metadata": {},
   "source": [
    "# Input data\n",
    "\n",
    "Provide your training and validation data in seperate pandas dataframes \n",
    "\n",
    "example shown below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c012178",
   "metadata": {},
   "source": [
    "**Modify the data loading part above as needed for your data**\n",
    "\n",
    "To run the training you need two dataframes (training and validation) each with the columns \"sequence\" and \"label\" and \"mask\"\n",
    "\n",
    "Columns are:\n",
    "+ protein sequence\n",
    "+ label is a list of len(protein sequence) with integers (from 0 to number of classes - 1) corresponding to predicted class at this position\n",
    "+ mask gives the possibility to ignore parts of the positions. Provide a list of len(protein sequence) where 1 is processed, while 0 is ignored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ea398922",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>sequence</th>\n",
       "      <th>mask</th>\n",
       "      <th>label</th>\n",
       "      <th>dataset</th>\n",
       "      <th>validation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1es5-A</td>\n",
       "      <td>VTKPTIAAVGGYAMNNGTGTTLYTKAADTRRSTGSTTKIMTAKVVL...</td>\n",
       "      <td>[0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>train</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2a6h-E</td>\n",
       "      <td>MAEPGIDKLFGMVDSKYRLTVVVAKRAQQLLRHGFKNTVLEPEERP...</td>\n",
       "      <td>[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, ...</td>\n",
       "      <td>train</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5b1a-P</td>\n",
       "      <td>MTHQTHAYHMVNPSPWPLTGALSALLMTSGLTMWFHFNSMTLLMIG...</td>\n",
       "      <td>[0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>train</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5ehi-C</td>\n",
       "      <td>GTGSQGETLGEKWKKKLNQLSRKEFDLYKKSGITEVDRTEAKEGLK...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, ...</td>\n",
       "      <td>train</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5egf-A</td>\n",
       "      <td>HHHHHHAVAKDSTESKSWEPFSLSPIKDPQALHAALCSKNVIPVTS...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>train</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     name                                           sequence  \\\n",
       "0  1es5-A  VTKPTIAAVGGYAMNNGTGTTLYTKAADTRRSTGSTTKIMTAKVVL...   \n",
       "1  2a6h-E  MAEPGIDKLFGMVDSKYRLTVVVAKRAQQLLRHGFKNTVLEPEERP...   \n",
       "2  5b1a-P  MTHQTHAYHMVNPSPWPLTGALSALLMTSGLTMWFHFNSMTLLMIG...   \n",
       "3  5ehi-C  GTGSQGETLGEKWKKKLNQLSRKEFDLYKKSGITEVDRTEAKEGLK...   \n",
       "4  5egf-A  HHHHHHAVAKDSTESKSWEPFSLSPIKDPQALHAALCSKNVIPVTS...   \n",
       "\n",
       "                                                mask  \\\n",
       "0  [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "1  [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "2  [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "3  [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...   \n",
       "\n",
       "                                               label dataset  validation  \n",
       "0  [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, ...   train       False  \n",
       "1  [0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, ...   train       False  \n",
       "2  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...   train       False  \n",
       "3  [0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, ...   train       False  \n",
       "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...   train       False  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For this example we import the secondary_structure dataset from https://github.com/J-SNACKKB/FLIP\n",
    "# For details, see publication here: https://openreview.net/forum?id=p2dMLEwL8tF\n",
    "import requests\n",
    "import zipfile\n",
    "from io import BytesIO\n",
    "from Bio import SeqIO\n",
    "import tempfile\n",
    "\n",
    "# Download the zip file from GitHub\n",
    "url = 'https://github.com/J-SNACKKB/FLIP/raw/main/splits/secondary_structure/splits.zip'\n",
    "\n",
    "response = requests.get(url)\n",
    "zip_file = zipfile.ZipFile(BytesIO(response.content))\n",
    "\n",
    "# Extract the fasta file to a temporary directory\n",
    "# Sequence File\n",
    "with tempfile.TemporaryDirectory() as temp_dir:\n",
    "    zip_file.extract('splits/sequences.fasta', temp_dir)\n",
    "\n",
    "    # Load the fasta files\n",
    "    fasta_file = open(temp_dir + '/splits/sequences.fasta')\n",
    "    \n",
    "    # Load FASTA file using Biopython\n",
    "    sequences = []\n",
    "    for record in SeqIO.parse(fasta_file, \"fasta\"):\n",
    "        sequences.append([record.name, str(record.seq)])\n",
    "\n",
    "    # Create dataframe\n",
    "    df = pd.DataFrame(sequences, columns=[\"name\", \"sequence\"])\n",
    "\n",
    "# Mask File\n",
    "with tempfile.TemporaryDirectory() as temp_dir:\n",
    "    zip_file.extract('splits/mask.fasta', temp_dir)\n",
    "\n",
    "    # Load the fasta files\n",
    "    fasta_file = open(temp_dir + '/splits/mask.fasta')\n",
    "    \n",
    "    # Load FASTA file using Biopython\n",
    "    sequences = []\n",
    "    for record in SeqIO.parse(fasta_file, \"fasta\"):\n",
    "        sequences.append([str(record.seq)])\n",
    "\n",
    "    # Add to dataframe\n",
    "    df = pd.concat([df, pd.DataFrame(sequences, columns=[\"mask\"])], axis=1) \n",
    "    \n",
    "# Label File\n",
    "with tempfile.TemporaryDirectory() as temp_dir:\n",
    "    zip_file.extract('splits/sampled.fasta', temp_dir)\n",
    "\n",
    "    # Load the fasta files\n",
    "    fasta_file = open(temp_dir + '/splits/sampled.fasta')\n",
    "    \n",
    "    # Load FASTA file using Biopython\n",
    "    sequences = []\n",
    "    for record in SeqIO.parse(fasta_file, \"fasta\"):\n",
    "\n",
    "        sequences.append([str(record.seq), record.description])\n",
    "\n",
    "    # Add to dataframe\n",
    "    df = pd.concat([df, pd.DataFrame(sequences, columns=[ \"label\", \"dataset\"])], axis=1)  \n",
    "\n",
    "# Get data split information\n",
    "df[\"validation\"]=df.dataset.str.split(\"=\").str[2]\n",
    "# str to bool\n",
    "df['validation'] = df['validation'].apply(lambda x: x == 'True')\n",
    "\n",
    "# Extract data split information\n",
    "df[\"dataset\"]=df.dataset.str.split(\"=\").str[1]\n",
    "df[\"dataset\"]=df.dataset.str.split(\" \").str[0]\n",
    "\n",
    "# Preprocess mask and label to lists\n",
    "# C is class 0, E is class 1, H is class 2\n",
    "df['label'] = df['label'].str.replace(\"C\",\"0\")\n",
    "df['label'] = df['label'].str.replace(\"E\",\"1\")\n",
    "df['label'] = df['label'].str.replace(\"H\",\"2\")\n",
    "\n",
    "# str to integer\n",
    "df['label'] = df['label'].apply(lambda x: [int(i) for i in x])\n",
    "df['mask'] = df['mask'].apply(lambda x: [int(i) for i in x])\n",
    "\n",
    "\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0a85fac6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Seperate test and train data \n",
    "my_test=df[df.dataset==\"test\"].reset_index(drop=True)\n",
    "df=df[df.dataset==\"train\"]\n",
    "\n",
    "# Get train and validation data\n",
    "my_train=df[df.validation!=True].reset_index(drop=True)\n",
    "my_valid=df[df.validation==True].reset_index(drop=True)\n",
    "\n",
    "# Drop unneeded columns\n",
    "my_train= my_train[[\"sequence\",\"label\",\"mask\"]]\n",
    "my_valid= my_valid[[\"sequence\",\"label\",\"mask\"]]\n",
    "my_test =  my_test[[\"sequence\",\"label\",\"mask\"]]\n",
    "\n",
    "# Set labels where mask == 0 to -100 (will be ignored by pytorch loss)\n",
    "my_train['label'] = my_train.apply(lambda row: [-100 if m == 0 else l for l, m in zip(row['label'], row['mask'])], axis=1)\n",
    "my_valid['label'] = my_valid.apply(lambda row: [-100 if m == 0 else l for l, m in zip(row['label'], row['mask'])], axis=1)\n",
    "my_test['label'] = my_test.apply(lambda row: [-100 if m == 0 else l for l, m in zip(row['label'], row['mask'])], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "14f14d83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "      <th>mask</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>VTKPTIAAVGGYAMNNGTGTTLYTKAADTRRSTGSTTKIMTAKVVL...</td>\n",
       "      <td>[-100, -100, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, ...</td>\n",
       "      <td>[0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MAEPGIDKLFGMVDSKYRLTVVVAKRAQQLLRHGFKNTVLEPEERP...</td>\n",
       "      <td>[-100, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 0, 0, ...</td>\n",
       "      <td>[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MTHQTHAYHMVNPSPWPLTGALSALLMTSGLTMWFHFNSMTLLMIG...</td>\n",
       "      <td>[-100, -100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>[0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GTGSQGETLGEKWKKKLNQLSRKEFDLYKKSGITEVDRTEAKEGLK...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, 0, 0, 2, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HHHHHHAVAKDSTESKSWEPFSLSPIKDPQALHAALCSKNVIPVTS...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence  \\\n",
       "0  VTKPTIAAVGGYAMNNGTGTTLYTKAADTRRSTGSTTKIMTAKVVL...   \n",
       "1  MAEPGIDKLFGMVDSKYRLTVVVAKRAQQLLRHGFKNTVLEPEERP...   \n",
       "2  MTHQTHAYHMVNPSPWPLTGALSALLMTSGLTMWFHFNSMTLLMIG...   \n",
       "3  GTGSQGETLGEKWKKKLNQLSRKEFDLYKKSGITEVDRTEAKEGLK...   \n",
       "4  HHHHHHAVAKDSTESKSWEPFSLSPIKDPQALHAALCSKNVIPVTS...   \n",
       "\n",
       "                                               label  \\\n",
       "0  [-100, -100, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, ...   \n",
       "1  [-100, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 0, 0, ...   \n",
       "2  [-100, -100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...   \n",
       "3  [-100, -100, -100, -100, -100, -100, 0, 0, 2, ...   \n",
       "4  [-100, -100, -100, -100, -100, -100, -100, -10...   \n",
       "\n",
       "                                                mask  \n",
       "0  [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "1  [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "2  [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "3  [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a71fc6d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "      <th>mask</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GMKVTNYQGATIDPYSKGLGMVPGTSIQLTDAARLEWNLLNEDVSL...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMPLDAGGQNSTQMVLAPGASIFRCRQCGQTISRRDWLLPMGGDHE...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MHFEAYPPEVNSANIYAGPGPDSMLAAARAWRSLDVEMTAVQRSFN...</td>\n",
       "      <td>[-100, 0, 2, 2, 2, 0, 0, 2, 2, 2, 2, 2, 2, 2, ...</td>\n",
       "      <td>[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MNIPTTTTKGEQAKSQLIAAALAQFGEYGLHATTRDIAALAGQNIA...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, 0, 2, 2, ...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GTSYENSLLVKQSGSLPLSSLTHVLRSLTPNARGIFRLLIKYQLDN...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence  \\\n",
       "0  GMKVTNYQGATIDPYSKGLGMVPGTSIQLTDAARLEWNLLNEDVSL...   \n",
       "1  AMPLDAGGQNSTQMVLAPGASIFRCRQCGQTISRRDWLLPMGGDHE...   \n",
       "2  MHFEAYPPEVNSANIYAGPGPDSMLAAARAWRSLDVEMTAVQRSFN...   \n",
       "3  MNIPTTTTKGEQAKSQLIAAALAQFGEYGLHATTRDIAALAGQNIA...   \n",
       "4  GTSYENSLLVKQSGSLPLSSLTHVLRSLTPNARGIFRLLIKYQLDN...   \n",
       "\n",
       "                                               label  \\\n",
       "0  [-100, -100, -100, -100, -100, -100, -100, -10...   \n",
       "1  [-100, -100, -100, -100, -100, -100, -100, -10...   \n",
       "2  [-100, 0, 2, 2, 2, 0, 0, 2, 2, 2, 2, 2, 2, 2, ...   \n",
       "3  [-100, -100, -100, -100, -100, -100, 0, 2, 2, ...   \n",
       "4  [-100, -100, -100, -100, -100, -100, -100, -10...   \n",
       "\n",
       "                                                mask  \n",
       "0  [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, ...  \n",
       "1  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "2  [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "3  [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \n",
       "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, ...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_valid.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b996723",
   "metadata": {},
   "source": [
    "# PT5 Model and Low Rank Adaptation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-standing",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## LoRA modification definition\n",
    "\n",
    "Implementation taken from https://github.com/r-three/t-few\n",
    "\n",
    "(https://github.com/r-three/t-few/blob/master/src/models/lora.py, https://github.com/r-three/t-few/tree/master/configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "separated-grenada",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Modifies an existing transformer and introduce the LoRA layers\n",
    "\n",
    "class LoRAConfig:\n",
    "    def __init__(self):\n",
    "        self.lora_rank = 4\n",
    "        self.lora_init_scale = 0.01\n",
    "        self.lora_modules = \".*SelfAttention|.*EncDecAttention\"\n",
    "        self.lora_layers = \"q|k|v|o\"\n",
    "        self.trainable_param_names = \".*layer_norm.*|.*lora_[ab].*\"\n",
    "        self.lora_scaling_rank = 1\n",
    "        # lora_modules and lora_layers are speicified with regular expressions\n",
    "        # see https://www.w3schools.com/python/python_regex.asp for reference\n",
    "        \n",
    "class LoRALinear(nn.Module):\n",
    "    def __init__(self, linear_layer, rank, scaling_rank, init_scale):\n",
    "        super().__init__()\n",
    "        self.in_features = linear_layer.in_features\n",
    "        self.out_features = linear_layer.out_features\n",
    "        self.rank = rank\n",
    "        self.scaling_rank = scaling_rank\n",
    "        self.weight = linear_layer.weight\n",
    "        self.bias = linear_layer.bias\n",
    "        if self.rank > 0:\n",
    "            self.lora_a = nn.Parameter(torch.randn(rank, linear_layer.in_features) * init_scale)\n",
    "            if init_scale < 0:\n",
    "                self.lora_b = nn.Parameter(torch.randn(linear_layer.out_features, rank) * init_scale)\n",
    "            else:\n",
    "                self.lora_b = nn.Parameter(torch.zeros(linear_layer.out_features, rank))\n",
    "        if self.scaling_rank:\n",
    "            self.multi_lora_a = nn.Parameter(\n",
    "                torch.ones(self.scaling_rank, linear_layer.in_features)\n",
    "                + torch.randn(self.scaling_rank, linear_layer.in_features) * init_scale\n",
    "            )\n",
    "            if init_scale < 0:\n",
    "                self.multi_lora_b = nn.Parameter(\n",
    "                    torch.ones(linear_layer.out_features, self.scaling_rank)\n",
    "                    + torch.randn(linear_layer.out_features, self.scaling_rank) * init_scale\n",
    "                )\n",
    "            else:\n",
    "                self.multi_lora_b = nn.Parameter(torch.ones(linear_layer.out_features, self.scaling_rank))\n",
    "\n",
    "    def forward(self, input):\n",
    "        if self.scaling_rank == 1 and self.rank == 0:\n",
    "            # parsimonious implementation for ia3 and lora scaling\n",
    "            if self.multi_lora_a.requires_grad:\n",
    "                hidden = F.linear((input * self.multi_lora_a.flatten()), self.weight, self.bias)\n",
    "            else:\n",
    "                hidden = F.linear(input, self.weight, self.bias)\n",
    "            if self.multi_lora_b.requires_grad:\n",
    "                hidden = hidden * self.multi_lora_b.flatten()\n",
    "            return hidden\n",
    "        else:\n",
    "            # general implementation for lora (adding and scaling)\n",
    "            weight = self.weight\n",
    "            if self.scaling_rank:\n",
    "                weight = weight * torch.matmul(self.multi_lora_b, self.multi_lora_a) / self.scaling_rank\n",
    "            if self.rank:\n",
    "                weight = weight + torch.matmul(self.lora_b, self.lora_a) / self.rank\n",
    "            return F.linear(input, weight, self.bias)\n",
    "\n",
    "    def extra_repr(self):\n",
    "        return \"in_features={}, out_features={}, bias={}, rank={}, scaling_rank={}\".format(\n",
    "            self.in_features, self.out_features, self.bias is not None, self.rank, self.scaling_rank\n",
    "        )\n",
    "\n",
    "\n",
    "def modify_with_lora(transformer, config):\n",
    "    for m_name, module in dict(transformer.named_modules()).items():\n",
    "        if re.fullmatch(config.lora_modules, m_name):\n",
    "            for c_name, layer in dict(module.named_children()).items():\n",
    "                if re.fullmatch(config.lora_layers, c_name):\n",
    "                    assert isinstance(\n",
    "                        layer, nn.Linear\n",
    "                    ), f\"LoRA can only be applied to torch.nn.Linear, but {layer} is {type(layer)}.\"\n",
    "                    setattr(\n",
    "                        module,\n",
    "                        c_name,\n",
    "                        LoRALinear(layer, config.lora_rank, config.lora_scaling_rank, config.lora_init_scale),\n",
    "                    )\n",
    "    return transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electronic-channels",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Classification model definition \n",
    "\n",
    "adding a token classification head on top of the encoder model\n",
    "\n",
    "modified from [EsmForTokenClassification](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/esm/modeling_esm.py#L1178)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "acting-archives",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class ClassConfig:\n",
    "    def __init__(self, dropout=0.2, num_labels=3):\n",
    "        self.dropout_rate = dropout\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "class T5EncoderForTokenClassification(T5PreTrainedModel):\n",
    "\n",
    "    def __init__(self, config: T5Config, class_config):\n",
    "        super().__init__(config)\n",
    "        self.num_labels = class_config.num_labels\n",
    "        self.config = config\n",
    "\n",
    "        self.shared = nn.Embedding(config.vocab_size, config.d_model)\n",
    "\n",
    "        encoder_config = copy.deepcopy(config)\n",
    "        encoder_config.use_cache = False\n",
    "        encoder_config.is_encoder_decoder = False\n",
    "        self.encoder = T5Stack(encoder_config, self.shared)\n",
    "\n",
    "        self.dropout = nn.Dropout(class_config.dropout_rate) \n",
    "        self.classifier = nn.Linear(config.hidden_size, class_config.num_labels)\n",
    "\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "        # Model parallel\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "\n",
    "    def parallelize(self, device_map=None):\n",
    "        self.device_map = (\n",
    "            get_device_map(len(self.encoder.block), range(torch.cuda.device_count()))\n",
    "            if device_map is None\n",
    "            else device_map\n",
    "        )\n",
    "        assert_device_map(self.device_map, len(self.encoder.block))\n",
    "        self.encoder.parallelize(self.device_map)\n",
    "        self.classifier = self.classifier.to(self.encoder.first_device)\n",
    "        self.model_parallel = True\n",
    "\n",
    "    def deparallelize(self):\n",
    "        self.encoder.deparallelize()\n",
    "        self.encoder = self.encoder.to(\"cpu\")\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "        torch.cuda.empty_cache()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.shared\n",
    "\n",
    "    def set_input_embeddings(self, new_embeddings):\n",
    "        self.shared = new_embeddings\n",
    "        self.encoder.set_input_embeddings(new_embeddings)\n",
    "\n",
    "    def get_encoder(self):\n",
    "        return self.encoder\n",
    "\n",
    "    def _prune_heads(self, heads_to_prune):\n",
    "        \"\"\"\n",
    "        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base\n",
    "        class PreTrainedModel\n",
    "        \"\"\"\n",
    "        for layer, heads in heads_to_prune.items():\n",
    "            self.encoder.layer[layer].attention.prune_heads(heads)\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids=None,\n",
    "        attention_mask=None,\n",
    "        head_mask=None,\n",
    "        inputs_embeds=None,\n",
    "        labels=None,\n",
    "        output_attentions=None,\n",
    "        output_hidden_states=None,\n",
    "        return_dict=None,\n",
    "    ):\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        outputs = self.encoder(\n",
    "            input_ids=input_ids,\n",
    "            attention_mask=attention_mask,\n",
    "            inputs_embeds=inputs_embeds,\n",
    "            head_mask=head_mask,\n",
    "            output_attentions=output_attentions,\n",
    "            output_hidden_states=output_hidden_states,\n",
    "            return_dict=return_dict,\n",
    "        )\n",
    "\n",
    "        sequence_output = outputs[0]\n",
    "        sequence_output = self.dropout(sequence_output)\n",
    "        logits = self.classifier(sequence_output)\n",
    "\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            loss_fct = CrossEntropyLoss()\n",
    "\n",
    "            active_loss = attention_mask.view(-1) == 1\n",
    "            active_logits = logits.view(-1, self.num_labels)\n",
    "\n",
    "            active_labels = torch.where(\n",
    "              active_loss, labels.view(-1), torch.tensor(-100).type_as(labels)\n",
    "            )\n",
    "\n",
    "            valid_logits=active_logits[active_labels!=-100]\n",
    "            valid_labels=active_labels[active_labels!=-100]\n",
    "            \n",
    "            valid_labels=valid_labels.type(torch.LongTensor).to('cuda:0')\n",
    "            \n",
    "            loss = loss_fct(valid_logits, valid_labels)\n",
    "            \n",
    "        \n",
    "        if not return_dict:\n",
    "            output = (logits,) + outputs[2:]\n",
    "            return ((loss,) + output) if loss is not None else output\n",
    "\n",
    "        return TokenClassifierOutput(\n",
    "            loss=loss,\n",
    "            logits=logits,\n",
    "            hidden_states=outputs.hidden_states,\n",
    "            attentions=outputs.attentions,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a0e217",
   "metadata": {},
   "source": [
    "## Modified ProtT5 model\n",
    "this creates a ProtT5 model with prediction head and LoRA modification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "split-austin",
   "metadata": {},
   "outputs": [],
   "source": [
    "def PT5_classification_model(num_labels):\n",
    "    # Load PT5 and tokenizer\n",
    "    model = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\")\n",
    "    tokenizer = T5Tokenizer.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\") \n",
    "    \n",
    "    # Create new Classifier model with PT5 dimensions\n",
    "    class_config=ClassConfig(num_labels=num_labels)\n",
    "    class_model=T5EncoderForTokenClassification(model.config,class_config)\n",
    "    \n",
    "    # Set encoder and embedding weights to checkpoint weights\n",
    "    class_model.shared=model.shared\n",
    "    class_model.encoder=model.encoder    \n",
    "    \n",
    "    # Delete the checkpoint model\n",
    "    model=class_model\n",
    "    del class_model\n",
    "    \n",
    "    # Print number of trainable parameters\n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_Classfier\\nTrainable Parameter: \"+ str(params))    \n",
    " \n",
    "    # Add model modification lora\n",
    "    config = LoRAConfig()\n",
    "    \n",
    "    # Add LoRA layers\n",
    "    model = modify_with_lora(model, config)\n",
    "    \n",
    "    # Freeze Embeddings and Encoder (except LoRA)\n",
    "    for (param_name, param) in model.shared.named_parameters():\n",
    "                param.requires_grad = False\n",
    "    for (param_name, param) in model.encoder.named_parameters():\n",
    "                param.requires_grad = False       \n",
    "\n",
    "    for (param_name, param) in model.named_parameters():\n",
    "            if re.fullmatch(config.trainable_param_names, param_name):\n",
    "                param.requires_grad = True\n",
    "\n",
    "    # Print trainable Parameter          \n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_LoRA_Classfier\\nTrainable Parameter: \"+ str(params) + \"\\n\")\n",
    "    \n",
    "    return model, tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beautiful-yeast",
   "metadata": {},
   "source": [
    "# Training Definition "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e735e819",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Deepspeed config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "eed91c2e",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Deepspeed config for optimizer CPU offload\n",
    "\n",
    "ds_config = {\n",
    "    \"fp16\": {\n",
    "        \"enabled\": \"auto\",\n",
    "        \"loss_scale\": 0,\n",
    "        \"loss_scale_window\": 1000,\n",
    "        \"initial_scale_power\": 16,\n",
    "        \"hysteresis\": 2,\n",
    "        \"min_loss_scale\": 1\n",
    "    },\n",
    "\n",
    "    \"optimizer\": {\n",
    "        \"type\": \"AdamW\",\n",
    "        \"params\": {\n",
    "            \"lr\": \"auto\",\n",
    "            \"betas\": \"auto\",\n",
    "            \"eps\": \"auto\",\n",
    "            \"weight_decay\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"scheduler\": {\n",
    "        \"type\": \"WarmupLR\",\n",
    "        \"params\": {\n",
    "            \"warmup_min_lr\": \"auto\",\n",
    "            \"warmup_max_lr\": \"auto\",\n",
    "            \"warmup_num_steps\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"zero_optimization\": {\n",
    "        \"stage\": 2,\n",
    "        \"offload_optimizer\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": True\n",
    "        },\n",
    "        \"allgather_partitions\": True,\n",
    "        \"allgather_bucket_size\": 2e8,\n",
    "        \"overlap_comm\": True,\n",
    "        \"reduce_scatter\": True,\n",
    "        \"reduce_bucket_size\": 2e8,\n",
    "        \"contiguous_gradients\": True\n",
    "    },\n",
    "\n",
    "    \"gradient_accumulation_steps\": \"auto\",\n",
    "    \"gradient_clipping\": \"auto\",\n",
    "    \"steps_per_print\": 2000,\n",
    "    \"train_batch_size\": \"auto\",\n",
    "    \"train_micro_batch_size_per_gpu\": \"auto\",\n",
    "    \"wall_clock_breakdown\": False\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92962861",
   "metadata": {},
   "source": [
    "## Training functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "liberal-learning",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seeds for reproducibility of your trainings run\n",
    "def set_seeds(s):\n",
    "    torch.manual_seed(s)\n",
    "    np.random.seed(s)\n",
    "    random.seed(s)\n",
    "    set_seed(s)\n",
    "\n",
    "# Dataset creation\n",
    "def create_dataset(tokenizer,seqs,labels):\n",
    "    tokenized = tokenizer(seqs, max_length=1024, padding=True, truncation=True)\n",
    "    dataset = Dataset.from_dict(tokenized)\n",
    "    # we need to cut of labels after 1023 positions for the data collator to add the correct padding (1023 + 1 special tokens)\n",
    "    labels = [l[:1023] for l in labels] \n",
    "    dataset = dataset.add_column(\"labels\", labels)\n",
    "     \n",
    "    return dataset\n",
    "    \n",
    "# Main training fuction\n",
    "def train_per_residue(\n",
    "        train_df,         #training data\n",
    "        valid_df,         #validation data      \n",
    "        num_labels= 3,    #number of classes\n",
    "    \n",
    "        # effective training batch size is batch * accum\n",
    "        # we recommend an effective batch size of 8 \n",
    "        batch= 4,         #for training\n",
    "        accum= 2,         #gradient accumulation\n",
    "    \n",
    "        val_batch = 16,   #batch size for evaluation\n",
    "        epochs= 10,       #training epochs\n",
    "        lr= 3e-4,         #recommended learning rate\n",
    "        seed= 42,         #random seed\n",
    "        deepspeed= True,  #if gpu is large enough disable deepspeed for training speedup\n",
    "        gpu= 1 ):         #gpu selection (1 for first gpu)\n",
    "\n",
    "    # Set gpu device\n",
    "    os.environ[\"CUDA_VISIBLE_DEVICES\"]=str(gpu-1)\n",
    "    \n",
    "    # Set all random seeds\n",
    "    set_seeds(seed)\n",
    "    \n",
    "    # load model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels)\n",
    "\n",
    "    # Preprocess inputs\n",
    "    # Replace uncommon AAs with \"X\"\n",
    "    train_df[\"sequence\"]=train_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    valid_df[\"sequence\"]=valid_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    # Add spaces between each amino acid for PT5 to correctly use them\n",
    "    train_df['sequence']=train_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "    valid_df['sequence']=valid_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "\n",
    "\n",
    "    # Create Datasets\n",
    "    train_set=create_dataset(tokenizer,list(train_df['sequence']),list(train_df['label']))\n",
    "    valid_set=create_dataset(tokenizer,list(valid_df['sequence']),list(valid_df['label']))\n",
    "\n",
    "    # Huggingface Trainer arguments\n",
    "    args = TrainingArguments(\n",
    "        \"./\",\n",
    "        evaluation_strategy = \"steps\",\n",
    "        eval_steps = 500,\n",
    "        logging_strategy = \"epoch\",\n",
    "        save_strategy = \"no\",\n",
    "        learning_rate=lr,\n",
    "        per_device_train_batch_size=batch,\n",
    "        #per_device_eval_batch_size=val_batch,\n",
    "        per_device_eval_batch_size=batch,\n",
    "        gradient_accumulation_steps=accum,\n",
    "        num_train_epochs=epochs,\n",
    "        seed = seed,\n",
    "        deepspeed= ds_config if deepspeed else None,\n",
    "    ) \n",
    "\n",
    "    # Metric definition for validation data\n",
    "    def compute_metrics(eval_pred):\n",
    "\n",
    "        metric = load(\"accuracy\")\n",
    "        predictions, labels = eval_pred\n",
    "        \n",
    "        labels = labels.reshape((-1,))\n",
    "        \n",
    "        predictions = np.argmax(predictions, axis=2)\n",
    "        predictions = predictions.reshape((-1,))\n",
    "        \n",
    "        predictions = predictions[labels!=-100]\n",
    "        labels = labels[labels!=-100]\n",
    "        \n",
    "        return metric.compute(predictions=predictions, references=labels)\n",
    "\n",
    "    # For token classification we need a data collator here to pad correctly\n",
    "    data_collator = DataCollatorForTokenClassification(tokenizer) \n",
    "\n",
    "    # Trainer          \n",
    "    trainer = Trainer(\n",
    "        model,\n",
    "        args,\n",
    "        train_dataset=train_set,\n",
    "        eval_dataset=valid_set,\n",
    "        tokenizer=tokenizer,\n",
    "        data_collator=data_collator,\n",
    "        compute_metrics=compute_metrics\n",
    "    )    \n",
    "    \n",
    "    # Train model\n",
    "    trainer.train()\n",
    "\n",
    "    return tokenizer, model, trainer.state.log_history\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac94ab1",
   "metadata": {},
   "source": [
    "# Run Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ede09d5",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "83fd6b5a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.embed_tokens.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.final_layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'lm_head.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1208144899\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 2510851\n",
      "\n",
      "[2023-07-14 07:40:25,122] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl\n",
      "[2023-07-14 07:40:25,307] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown\n",
      "[2023-07-14 07:40:27,289] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Detected CUDA files, patching ldflags\n",
      "Emitting ninja build file /homes/user/.cache/torch_extensions/py39_cu117/cpu_adam/build.ninja...\n",
      "Building extension module cpu_adam...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module cpu_adam...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load cpu_adam op: 2.938427209854126 seconds\n",
      "[2023-07-14 07:40:33,044] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer\n",
      "[2023-07-14 07:40:33,081] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam\n",
      "[2023-07-14 07:40:33,082] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>\n",
      "[2023-07-14 07:40:33,082] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer\n",
      "[2023-07-14 07:40:33,082] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 200000000\n",
      "[2023-07-14 07:40:33,083] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 200000000\n",
      "[2023-07-14 07:40:33,083] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True\n",
      "[2023-07-14 07:40:33,084] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Emitting ninja build file /homes/user/.cache/torch_extensions/py39_cu117/utils/build.ninja...\n",
      "Building extension module utils...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module utils...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load utils op: 0.47357940673828125 seconds\n",
      "Rank: 0 partition count [1] and sizes[(2510852, False)] \n",
      "[2023-07-14 07:40:33,828] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states\n",
      "[2023-07-14 07:40:33,829] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-07-14 07:40:33,830] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 47.34 GB, percent = 25.4%\n",
      "[2023-07-14 07:40:34,032] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states\n",
      "[2023-07-14 07:40:34,033] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-07-14 07:40:34,034] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 47.37 GB, percent = 25.4%\n",
      "[2023-07-14 07:40:34,035] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized\n",
      "[2023-07-14 07:40:34,229] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer\n",
      "[2023-07-14 07:40:34,230] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-07-14 07:40:34,231] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 47.37 GB, percent = 25.4%\n",
      "[2023-07-14 07:40:34,242] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw\n",
      "[2023-07-14 07:40:34,242] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR\n",
      "[2023-07-14 07:40:34,243] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fe65c115880>\n",
      "[2023-07-14 07:40:34,243] [INFO] [logging.py:75:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-14 07:40:34,244] [INFO] [config.py:1009:print] DeepSpeedEngine configuration:\n",
      "[2023-07-14 07:40:34,245] [INFO] [config.py:1013:print]   activation_checkpointing_config  {\n",
      "    \"partition_activations\": false, \n",
      "    \"contiguous_memory_optimization\": false, \n",
      "    \"cpu_checkpointing\": false, \n",
      "    \"number_checkpoints\": null, \n",
      "    \"synchronize_checkpoint_boundary\": false, \n",
      "    \"profile\": false\n",
      "}\n",
      "[2023-07-14 07:40:34,245] [INFO] [config.py:1013:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}\n",
      "[2023-07-14 07:40:34,246] [INFO] [config.py:1013:print]   amp_enabled .................. False\n",
      "[2023-07-14 07:40:34,246] [INFO] [config.py:1013:print]   amp_params ................... False\n",
      "[2023-07-14 07:40:34,247] [INFO] [config.py:1013:print]   autotuning_config ............ {\n",
      "    \"enabled\": false, \n",
      "    \"start_step\": null, \n",
      "    \"end_step\": null, \n",
      "    \"metric_path\": null, \n",
      "    \"arg_mappings\": null, \n",
      "    \"metric\": \"throughput\", \n",
      "    \"model_info\": null, \n",
      "    \"results_dir\": \"autotuning_results\", \n",
      "    \"exps_dir\": \"autotuning_exps\", \n",
      "    \"overwrite\": true, \n",
      "    \"fast\": true, \n",
      "    \"start_profile_step\": 3, \n",
      "    \"end_profile_step\": 5, \n",
      "    \"tuner_type\": \"gridsearch\", \n",
      "    \"tuner_early_stopping\": 5, \n",
      "    \"tuner_num_trials\": 50, \n",
      "    \"model_info_path\": null, \n",
      "    \"mp_size\": 1, \n",
      "    \"max_train_batch_size\": null, \n",
      "    \"min_train_batch_size\": 1, \n",
      "    \"max_train_micro_batch_size_per_gpu\": 1.024000e+03, \n",
      "    \"min_train_micro_batch_size_per_gpu\": 1, \n",
      "    \"num_tuning_micro_batch_sizes\": 3\n",
      "}\n",
      "[2023-07-14 07:40:34,247] [INFO] [config.py:1013:print]   bfloat16_enabled ............. False\n",
      "[2023-07-14 07:40:34,248] [INFO] [config.py:1013:print]   checkpoint_parallel_write_pipeline  False\n",
      "[2023-07-14 07:40:34,248] [INFO] [config.py:1013:print]   checkpoint_tag_validation_enabled  True\n",
      "[2023-07-14 07:40:34,248] [INFO] [config.py:1013:print]   checkpoint_tag_validation_fail  False\n",
      "[2023-07-14 07:40:34,249] [INFO] [config.py:1013:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fe65a8d58b0>\n",
      "[2023-07-14 07:40:34,249] [INFO] [config.py:1013:print]   communication_data_type ...... None\n",
      "[2023-07-14 07:40:34,249] [INFO] [config.py:1013:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}\n",
      "[2023-07-14 07:40:34,250] [INFO] [config.py:1013:print]   curriculum_enabled_legacy .... False\n",
      "[2023-07-14 07:40:34,250] [INFO] [config.py:1013:print]   curriculum_params_legacy ..... False\n",
      "[2023-07-14 07:40:34,250] [INFO] [config.py:1013:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}\n",
      "[2023-07-14 07:40:34,251] [INFO] [config.py:1013:print]   data_efficiency_enabled ...... False\n",
      "[2023-07-14 07:40:34,251] [INFO] [config.py:1013:print]   dataloader_drop_last ......... False\n",
      "[2023-07-14 07:40:34,252] [INFO] [config.py:1013:print]   disable_allgather ............ False\n",
      "[2023-07-14 07:40:34,252] [INFO] [config.py:1013:print]   dump_state ................... False\n",
      "[2023-07-14 07:40:34,252] [INFO] [config.py:1013:print]   dynamic_loss_scale_args ...... None\n",
      "[2023-07-14 07:40:34,253] [INFO] [config.py:1013:print]   eigenvalue_enabled ........... False\n",
      "[2023-07-14 07:40:34,253] [INFO] [config.py:1013:print]   eigenvalue_gas_boundary_resolution  1\n",
      "[2023-07-14 07:40:34,253] [INFO] [config.py:1013:print]   eigenvalue_layer_name ........ bert.encoder.layer\n",
      "[2023-07-14 07:40:34,254] [INFO] [config.py:1013:print]   eigenvalue_layer_num ......... 0\n",
      "[2023-07-14 07:40:34,254] [INFO] [config.py:1013:print]   eigenvalue_max_iter .......... 100\n",
      "[2023-07-14 07:40:34,254] [INFO] [config.py:1013:print]   eigenvalue_stability ......... 1e-06\n",
      "[2023-07-14 07:40:34,255] [INFO] [config.py:1013:print]   eigenvalue_tol ............... 0.01\n",
      "[2023-07-14 07:40:34,255] [INFO] [config.py:1013:print]   eigenvalue_verbose ........... False\n",
      "[2023-07-14 07:40:34,256] [INFO] [config.py:1013:print]   elasticity_enabled ........... False\n",
      "[2023-07-14 07:40:34,256] [INFO] [config.py:1013:print]   flops_profiler_config ........ {\n",
      "    \"enabled\": false, \n",
      "    \"profile_step\": 1, \n",
      "    \"module_depth\": -1, \n",
      "    \"top_modules\": 1, \n",
      "    \"detailed\": true, \n",
      "    \"output_file\": null\n",
      "}\n",
      "[2023-07-14 07:40:34,256] [INFO] [config.py:1013:print]   fp16_auto_cast ............... None\n",
      "[2023-07-14 07:40:34,257] [INFO] [config.py:1013:print]   fp16_enabled ................. False\n",
      "[2023-07-14 07:40:34,257] [INFO] [config.py:1013:print]   fp16_master_weights_and_gradients  False\n",
      "[2023-07-14 07:40:34,257] [INFO] [config.py:1013:print]   global_rank .................. 0\n",
      "[2023-07-14 07:40:34,258] [INFO] [config.py:1013:print]   grad_accum_dtype ............. None\n",
      "[2023-07-14 07:40:34,258] [INFO] [config.py:1013:print]   gradient_accumulation_steps .. 1\n",
      "[2023-07-14 07:40:34,258] [INFO] [config.py:1013:print]   gradient_clipping ............ 1.0\n",
      "[2023-07-14 07:40:34,259] [INFO] [config.py:1013:print]   gradient_predivide_factor .... 1.0\n",
      "[2023-07-14 07:40:34,259] [INFO] [config.py:1013:print]   initial_dynamic_scale ........ 65536\n",
      "[2023-07-14 07:40:34,260] [INFO] [config.py:1013:print]   load_universal_checkpoint .... False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-14 07:40:34,260] [INFO] [config.py:1013:print]   loss_scale ................... 0\n",
      "[2023-07-14 07:40:34,260] [INFO] [config.py:1013:print]   memory_breakdown ............. False\n",
      "[2023-07-14 07:40:34,261] [INFO] [config.py:1013:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False\n",
      "[2023-07-14 07:40:34,261] [INFO] [config.py:1013:print]   nebula_config ................ {\n",
      "    \"enabled\": false, \n",
      "    \"persistent_storage_path\": null, \n",
      "    \"persistent_time_interval\": 100, \n",
      "    \"num_of_version_in_retention\": 2, \n",
      "    \"enable_nebula_load\": true, \n",
      "    \"load_path\": null\n",
      "}\n",
      "[2023-07-14 07:40:34,261] [INFO] [config.py:1013:print]   optimizer_legacy_fusion ...... False\n",
      "[2023-07-14 07:40:34,262] [INFO] [config.py:1013:print]   optimizer_name ............... adamw\n",
      "[2023-07-14 07:40:34,262] [INFO] [config.py:1013:print]   optimizer_params ............. {'lr': 0.0003, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}\n",
      "[2023-07-14 07:40:34,263] [INFO] [config.py:1013:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}\n",
      "[2023-07-14 07:40:34,263] [INFO] [config.py:1013:print]   pld_enabled .................. False\n",
      "[2023-07-14 07:40:34,263] [INFO] [config.py:1013:print]   pld_params ................... False\n",
      "[2023-07-14 07:40:34,264] [INFO] [config.py:1013:print]   prescale_gradients ........... False\n",
      "[2023-07-14 07:40:34,264] [INFO] [config.py:1013:print]   scheduler_name ............... WarmupLR\n",
      "[2023-07-14 07:40:34,264] [INFO] [config.py:1013:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 0.0003, 'warmup_num_steps': 0}\n",
      "[2023-07-14 07:40:34,265] [INFO] [config.py:1013:print]   sparse_attention ............. None\n",
      "[2023-07-14 07:40:34,265] [INFO] [config.py:1013:print]   sparse_gradients_enabled ..... False\n",
      "[2023-07-14 07:40:34,265] [INFO] [config.py:1013:print]   steps_per_print .............. 2000\n",
      "[2023-07-14 07:40:34,266] [INFO] [config.py:1013:print]   train_batch_size ............. 1\n",
      "[2023-07-14 07:40:34,266] [INFO] [config.py:1013:print]   train_micro_batch_size_per_gpu  1\n",
      "[2023-07-14 07:40:34,266] [INFO] [config.py:1013:print]   use_node_local_storage ....... False\n",
      "[2023-07-14 07:40:34,267] [INFO] [config.py:1013:print]   wall_clock_breakdown ......... False\n",
      "[2023-07-14 07:40:34,267] [INFO] [config.py:1013:print]   world_size ................... 1\n",
      "[2023-07-14 07:40:34,269] [INFO] [config.py:1013:print]   zero_allow_untested_optimizer  False\n",
      "[2023-07-14 07:40:34,270] [INFO] [config.py:1013:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False\n",
      "[2023-07-14 07:40:34,270] [INFO] [config.py:1013:print]   zero_enabled ................. True\n",
      "[2023-07-14 07:40:34,270] [INFO] [config.py:1013:print]   zero_optimization_stage ...... 2\n",
      "[2023-07-14 07:40:34,271] [INFO] [config.py:998:print_user_config]   json = {\n",
      "    \"fp16\": {\n",
      "        \"enabled\": false, \n",
      "        \"loss_scale\": 0, \n",
      "        \"loss_scale_window\": 1000, \n",
      "        \"initial_scale_power\": 16, \n",
      "        \"hysteresis\": 2, \n",
      "        \"min_loss_scale\": 1\n",
      "    }, \n",
      "    \"optimizer\": {\n",
      "        \"type\": \"AdamW\", \n",
      "        \"params\": {\n",
      "            \"lr\": 0.0003, \n",
      "            \"betas\": [0.9, 0.999], \n",
      "            \"eps\": 1e-08, \n",
      "            \"weight_decay\": 0.0\n",
      "        }\n",
      "    }, \n",
      "    \"scheduler\": {\n",
      "        \"type\": \"WarmupLR\", \n",
      "        \"params\": {\n",
      "            \"warmup_min_lr\": 0, \n",
      "            \"warmup_max_lr\": 0.0003, \n",
      "            \"warmup_num_steps\": 0\n",
      "        }\n",
      "    }, \n",
      "    \"zero_optimization\": {\n",
      "        \"stage\": 2, \n",
      "        \"offload_optimizer\": {\n",
      "            \"device\": \"cpu\", \n",
      "            \"pin_memory\": true\n",
      "        }, \n",
      "        \"allgather_partitions\": true, \n",
      "        \"allgather_bucket_size\": 2.000000e+08, \n",
      "        \"overlap_comm\": true, \n",
      "        \"reduce_scatter\": true, \n",
      "        \"reduce_bucket_size\": 2.000000e+08, \n",
      "        \"contiguous_gradients\": true\n",
      "    }, \n",
      "    \"gradient_accumulation_steps\": 1, \n",
      "    \"gradient_clipping\": 1.0, \n",
      "    \"steps_per_print\": 2.000000e+03, \n",
      "    \"train_batch_size\": 1, \n",
      "    \"train_micro_batch_size_per_gpu\": 1, \n",
      "    \"wall_clock_breakdown\": false\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "No modifications detected for re-loaded extension module utils, skipping build step...\n",
      "Loading extension module utils...\n",
      "***** Running training *****\n",
      "  Num examples = 9712\n",
      "  Num Epochs = 1\n",
      "  Instantaneous batch size per device = 1\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 1\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 9712\n",
      "  Number of trainable parameters = 2510851\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load utils op: 0.002664327621459961 seconds\n",
      "Adam Optimizer #0 is created with AVX2 arithmetic capability.\n",
      "Config: alpha=0.000300, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='9712' max='9712' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [9712/9712 3:42:26, Epoch 1/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.387572</td>\n",
       "      <td>0.846588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.361559</td>\n",
       "      <td>0.853488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.353014</td>\n",
       "      <td>0.856071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.352000</td>\n",
       "      <td>0.855998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.346579</td>\n",
       "      <td>0.859100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.339916</td>\n",
       "      <td>0.861003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.340077</td>\n",
       "      <td>0.861015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.337155</td>\n",
       "      <td>0.862594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.333077</td>\n",
       "      <td>0.863735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.332154</td>\n",
       "      <td>0.864162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.328296</td>\n",
       "      <td>0.864738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.328190</td>\n",
       "      <td>0.865184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.327117</td>\n",
       "      <td>0.866580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.330076</td>\n",
       "      <td>0.864112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.322919</td>\n",
       "      <td>0.866741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.328622</td>\n",
       "      <td>0.865287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.321395</td>\n",
       "      <td>0.867443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9000</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.321478</td>\n",
       "      <td>0.867290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9500</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.317662</td>\n",
       "      <td>0.868381</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-14 08:21:19,519] [INFO] [logging.py:75:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-14 08:21:19,520] [INFO] [timer.py:198:stop] epoch=0/micro_step=2000/global_step=2000, RunningAvgSamplesPerSec=1.3915422776812985, CurrSamplesPerSec=1.3891091666611468, MemAllocated=4.51GB, MaxMemAllocated=18.45GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-14 09:07:38,015] [INFO] [logging.py:75:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-14 09:07:38,016] [INFO] [timer.py:198:stop] epoch=0/micro_step=4000/global_step=4000, RunningAvgSamplesPerSec=1.3912934587099546, CurrSamplesPerSec=1.3904911751318367, MemAllocated=4.51GB, MaxMemAllocated=18.45GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-14 09:53:55,728] [INFO] [logging.py:75:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-14 09:53:55,729] [INFO] [timer.py:198:stop] epoch=0/micro_step=6000/global_step=6000, RunningAvgSamplesPerSec=1.3911471224627971, CurrSamplesPerSec=1.3927013659039351, MemAllocated=4.51GB, MaxMemAllocated=18.45GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-14 10:40:13,016] [INFO] [logging.py:75:log_dist] [Rank 0] step=8000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-14 10:40:13,017] [INFO] [timer.py:198:stop] epoch=0/micro_step=8000/global_step=8000, RunningAvgSamplesPerSec=1.3911751523253517, CurrSamplesPerSec=1.391493132447341, MemAllocated=4.51GB, MaxMemAllocated=18.45GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1080\n",
      "  Batch size = 1\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model, history = train_per_residue(my_train, my_valid, num_labels=3, batch=1, accum=1, epochs=1, seed=42, gpu=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bab485",
   "metadata": {},
   "source": [
    "## Plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f8465267",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4oAAAHWCAYAAAAxXnddAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAACGMElEQVR4nOzdd1yV5f/H8dc5bBAHIuBAcSsOcJJapoZhmSMbaJZKO80GNjRzpCVlX81S07JMM1eZWj8zzVArRznJcqDmQFRwL1RQOL8/7jiJogICN+P9fDzuB+dc577v87nhlLy5rvu6LDabzYaIiIiIiIjIv6xmFyAiIiIiIiIFi4KiiIiIiIiIZKCgKCIiIiIiIhkoKIqIiIiIiEgGCooiIiIiIiKSgYKiiIiIiIiIZKCgKCIiIiIiIhkoKIqIiIiIiEgGCooiIiIiIiKSgYKiiIiYrk+fPgQEBOTo2OHDh2OxWHK3oDyyb98+LBYL06ZNM7sUERGRG1JQFBGR67JYLFnaVq5caXappujTpw8lSpS47usWi4Xnn3/+lt/n448/VrgUEZF85Wh2ASIiUnDNmDEjw/Mvv/ySZcuWXdNet27dW3qfKVOmkJaWlqNj33zzTQYOHHhL759fqlSpwoULF3BycsrWcR9//DHe3t706dMnbwoTERG5ioKiiIhc16OPPprh+e+//86yZcuuab/a+fPncXd3z/L7ZDc4XcnR0RFHx8Lxz5nFYsHV1dXsMgC4ePEizs7OWK0aXCQiItfSvw4iInJL2rRpQ/369dm4cSOtW7fG3d2dN954A4DvvvuOjh07UqFCBVxcXKhevTojR44kNTU1wzmuvkcx/V6+//3vf3z66adUr14dFxcXmjVrxvr16zMcm9k9iulDPhcuXEj9+vVxcXGhXr16LFmy5Jr6V65cSdOmTXF1daV69ep88skneXbfY2b3KCYkJBAREUGlSpVwcXGhfPnydOnShX379gEQEBDA1q1b+eWXX+xDfdu0aWM/fs+ePTz00EN4eXnh7u7Obbfdxg8//HDNNVosFubMmcObb75JxYoVcXd3JyYmBovFwgcffHBNrWvWrMFisTB79uxc/z6IiEjBVzj+BCsiIgXa8ePHueeee+jevTuPPvoovr6+AEybNo0SJUoQGRlJiRIlWL58OUOHDuXMmTO8//77Nz3vrFmzOHv2LM888wwWi4XRo0fTrVs39uzZc9NeyFWrVjF//nz69u2Lp6cnH330EQ888ABxcXGULVsWgM2bN9OhQwfKly/PW2+9RWpqKiNGjKBcuXLZuv5jx45la/8rPfDAA2zdupX+/fsTEBDAkSNHWLZsGXFxcQQEBDBu3Dj69+9PiRIlGDx4MID9+5uYmEjLli05f/48L7zwAmXLlmX69Ol07tyZefPmcf/992d4r5EjR+Ls7Mwrr7xCcnIyderUoVWrVsycOZOXX345w74zZ87E09OTLl265PjaRESkELOJiIhkUb9+/WxX/9Nx55132gDb5MmTr9n//Pnz17Q988wzNnd3d9vFixftbb1797ZVqVLF/nzv3r02wFa2bFnbiRMn7O3fffedDbD93//9n71t2LBh19QE2JydnW27d++2t/355582wDZ+/Hh7W6dOnWzu7u62gwcP2tt27dplc3R0vOacmendu7cNuOHWr1+/a67riy++sNlsNtvJkydtgO3999+/4fvUq1fPduedd17T/tJLL9kA22+//WZvO3v2rK1q1aq2gIAAW2pqqs1ms9lWrFhhA2zVqlW75mfyySef2ADb9u3b7W0pKSk2b29vW+/evW/6PRARkaJJQ09FROSWubi4EBERcU27m5ub/fHZs2c5duwYd9xxB+fPn2fHjh03PW94eDhlypSxP7/jjjsAY7jlzYSGhlK9enX784YNG1KyZEn7sampqfz888907dqVChUq2PerUaMG99xzz03Pn87V1ZVly5Zlut2Mm5sbzs7OrFy5kpMnT2b5PdMtXryY5s2bc/vtt9vbSpQowdNPP82+ffvYtm1bhv179+6d4WcC8PDDD+Pq6srMmTPtbUuXLuXYsWM3vRdVRESKLg09FRGRW1axYkWcnZ2vad+6dStvvvkmy5cv58yZMxleO3369E3PW7ly5QzP00NjVkLV1cemH59+7JEjR7hw4QI1atS4Zr/M2q7HwcGB0NDQLO9/JRcXF9577z0GDBiAr68vt912G/fddx+9evXCz8/vpsfv37+fkJCQa9rTZ6Hdv38/9evXt7dXrVr1mn1Lly5Np06dmDVrFiNHjgSMYacVK1akXbt2ObouEREp/NSjKCIit+zqXiqAU6dOceedd/Lnn38yYsQI/u///o9ly5bx3nvvAWRpOQwHB4dM2202W54em59eeukldu7cSVRUFK6urgwZMoS6deuyefPmXH+vzH5OAL169WLPnj2sWbOGs2fP8v3339OjRw/NiCoiUoypR1FERPLEypUrOX78OPPnz6d169b29r1795pY1X98fHxwdXVl9+7d17yWWVteql69OgMGDGDAgAHs2rWL4OBgxowZw1dffQVw3RlYq1SpQmxs7DXt6cN6q1SpkqX379ChA+XKlWPmzJmEhIRw/vx5HnvssRxejYiIFAX6U6GIiOSJ9B69K3vwUlJS+Pjjj80qKYP0IaMLFy7k0KFD9vbdu3fz448/5ksN58+f5+LFixnaqlevjqenJ8nJyfY2Dw8PTp06dc3x9957L+vWrWPt2rX2tqSkJD799FMCAgIIDAzMUh2Ojo706NGDr7/+mmnTptGgQQMaNmyYs4sSEZEiQT2KIiKSJ1q2bEmZMmXo3bs3L7zwAhaLhRkzZhSooZ/Dhw/np59+olWrVjz33HOkpqYyYcIE6tevT0xMTJ6//86dO7nrrrt4+OGHCQwMxNHRkQULFpCYmEj37t3t+zVp0oRJkybx9ttvU6NGDXx8fGjXrh0DBw5k9uzZ3HPPPbzwwgt4eXkxffp09u7dy7fffputoaO9evXio48+YsWKFfbhwSIiUnwpKIqISJ4oW7YsixYtYsCAAbz55puUKVOGRx99lLvuuouwsDCzywOMAPbjjz/yyiuvMGTIEPz9/RkxYgTbt2/P0qyst8rf358ePXoQHR3NjBkzcHR0pE6dOnz99dc88MAD9v2GDh3K/v37GT16NGfPnuXOO++kXbt2+Pr6smbNGl5//XXGjx/PxYsXadiwIf/3f/9Hx44ds1VLkyZNqFevHtu3b6dnz565fakiIlLIWGwF6U+7IiIiBUDXrl3ZunUru3btMruUfNWoUSO8vLyIjo42uxQRETGZ7lEUEZFi7cKFCxme79q1i8WLF9OmTRtzCjLJhg0biImJoVevXmaXIiIiBYB6FEVEpFgrX748ffr0oVq1auzfv59JkyaRnJzM5s2bqVmzptnl5bm///6bjRs3MmbMGI4dO8aePXtwdXU1uywRETGZ7lEUEZFirUOHDsyePZuEhARcXFxo0aIFo0aNKhYhEWDevHmMGDGC2rVrM3v2bIVEEREB1KMoIiIiIiKSa3799Vfef/99Nm7cyOHDh1mwYAFdu3a94TErV64kMjKSrVu34u/vz5tvvkmfPn3ypd7r0T2KIiIiIiIiuSQpKYmgoCAmTpyYpf337t1Lx44dadu2LTExMbz00ks8+eSTLF26NI8rvTH1KIqIiIiIiOQBi8Vy0x7F119/nR9++IG///7b3ta9e3dOnTrFkiVL8qHKzOkexUxcvnyZzZs34+vrm63FikVEREREpGhJS0sjLi6OwMBAHB3/i08uLi64uLjc8vnXrl1LaGhohrawsDBeeumlWz73rVBQzMTmzZtp3ry52WWIiIiIiEgBNWzYMIYPH37L50lISMDX1zdDm6+vL2fOnOHChQu4ubnd8nvkhOlBceLEibz//vskJCQQFBTE+PHjrxvS5s+fz6hRo9i9ezeXLl2iZs2aDBgwgMcee8y+T2JiIq+//jo//fQTp06donXr1owfPz5bs9el/6DWrVtH+fLlb+0CRURERESk0Dp8+DDNmzfn77//xt/f396eG72JBZmpQXHu3LlERkYyefJkQkJCGDduHGFhYcTGxuLj43PN/l5eXgwePJg6derg7OzMokWLiIiIwMfHh7CwMGw2G127dsXJyYnvvvuOkiVLMnbsWEJDQ9m2bRseHh5Zqit9uGn58uWpVKlSrl6ziIiIiIgUPqVKlaJkyZK5fl4/Pz8SExMztCUmJlKyZEnTehPB5FlPx44dy1NPPUVERASBgYFMnjwZd3d3pk6dmun+bdq04f7776du3bpUr16dF198kYYNG7Jq1SoAdu3axe+//86kSZNo1qwZtWvXZtKkSVy4cIHZs2fn56WJiIiIiIjcVIsWLYiOjs7QtmzZMlq0aGFSRQbTgmJKSgobN27McOOm1WolNDSUtWvX3vR4m81GdHQ0sbGxtG7dGoDk5GSADIsFW61WXFxc7GEyM8nJyZw5c8a+nT17NqeXJSIiIiIixdi5c+eIiYkhJiYGMJa/iImJIS4uDoBBgwbRq1cv+/7PPvsse/bs4bXXXmPHjh18/PHHfP3117z88stmlG9nWlA8duwYqampmd64mZCQcN3jTp8+TYkSJXB2dqZjx46MHz+e9u3bA1CnTh0qV67MoEGDOHnyJCkpKbz33nvEx8dz+PDh654zKiqKUqVK2bfAwMDcuUgRERERESlWNmzYQKNGjWjUqBEAkZGRNGrUiKFDhwLGPY/poRGgatWq/PDDDyxbtoygoCDGjBnDZ599RlhYmCn1pzN9Mpvs8vT0JCYmhnPnzhEdHU1kZCTVqlWjTZs2ODk5MX/+fJ544gm8vLxwcHAgNDSUe+65hxstFzlo0CAiIyPtzw8ePKiwKCIiIiIi2damTZsbZo9p06ZleszmzZvzsKrsMy0oent74+DgkOmNm35+ftc9zmq1UqNGDQCCg4PZvn07UVFRtGnTBoAmTZoQExPD6dOnSUlJoVy5coSEhNC0adPrnvPqNVDOnDlzC1cmIiIiIiJSuJk29NTZ2ZkmTZpkuHEzLS2N6OjobN24mZaWZr838UqlSpWiXLly7Nq1iw0bNtClS5dcqVtERERERKSoM3XoaWRkJL1796Zp06Y0b96ccePGkZSUREREBAC9evWiYsWKREVFAca9hE2bNqV69eokJyezePFiZsyYwaRJk+zn/OabbyhXrhyVK1fmr7/+4sUXX6Rr167cfffdplyjiIiIiIhIYWNqUAwPD+fo0aMMHTqUhIQEgoODWbJkiX2Cm7i4OPuahgBJSUn07duX+Ph43NzcqFOnDl999RXh4eH2fQ4fPkxkZCSJiYmUL1+eXr16MWTIkHy/NhERERERkcLKYrvRnZbFVHx8PP7+/hw4cIBKlSqZXY6IiIiIiJikuGYD0+5RFBERERERkYJJQVFEREREREQyUFAUERERERGRDBQUJfvS0syuQERERERE8pCComTdqnEwriH89bXZlYiIiIiISB5SUJSsu3gKTu2Hf5abXYmIiIiIiOQhBUXJumptja97VoJWVRERERERKbIUFCXrKt8Gjm5wLhGObDO7GhERERERySMKipJ1ji4Q0Mp4/M8Kc2sREREREZE8o6Ao2ZM+/FT3KYqIiIiIFFkKipI91f8NivvXwKWL5tYiIiIiIiJ5QkFRsscnEEr4wuULcOAPs6sREREREZE8oKAo2WOxXDH7qe5TFBEREREpihQUJfuq6z5FEREREZGiTEFRsq9aG+Pr4S2QdNzUUkREREREJPcpKEr2efqBTz3ABntXml2NiIiIiIjkMgVFyRn78FPdpygiIiIiUtQoKErOVLsiKNps5tYiIiIiIiK5SkFRcqZKS3BwhjPxcHy32dWIiIiIiEguUlCUnHF2h8q3GY81/FREREREpEhRUJSc03qKIiIiIiJFkoKi5Fz6hDZ7f4PUS+bWIiIiIiIiuUZBUXLOLwjcvCDlLMRvMLsaERERERHJJQqKknNWK1RrYzzW8FMRERERkSJDQVFujdZTFBEREREpchQU5dakT2hzcANcOGVqKSIiIiIikjsUFOXWlPaHsjXAlgb7fjO7GhERERERyQUKinLrqrczvmr4qYiIiIhIkaCgKLdO6ymKiIiIiBQpCopy6wJuB4sDnNgDJ/eZXY2IiIiIiNwiBUW5da4loVIz47GGn4qIiIiIFHoKipI70u9T1PBTEREREZFCT0FRckf6eop7foG0VHNrERERERGRW6KgKLmjQmNwKQUXT8GhGLOrERERERGRW6CgKLnDwRGq3mE83rPc3FpEREREROSWKChK7kkffvrPSlPLEBERERGRW6OgKLknfT3FA39A8jlzaxERERERkRxzNLsAKUK8qkHpynAqDvavhlphZlckIiIi2ZWWZny1FsP+BJsNLl+GS5eyv6VeMZmfxZLxa2ZtN3otu/tn9lpq6vW3y5dv/PrNtuweX7cuDBmSvZ+FmE5BUXKPxWL0Km6abqynqKAoIkWFzZZxS0u7/vO8fpzbx+f387Q045fMG23pv4hmdcvp/mlp4OwMLi7X3272em7t5+wMKSmQnAwXLxpfr/f4Zq/f6jnSA4/VCo6OOd+cnG7t+PRzpKXlLLjlZLt82dz/1xRVd96poFgImR4UJ06cyPvvv09CQgJBQUGMHz+e5s2bZ7rv/PnzGTVqFLt37+bSpUvUrFmTAQMG8Nhjj9n3OXfuHAMHDmThwoUcP36cqlWr8sILL/Dss8/m1yUVb9XbGUFR6ymK5I/0X7zTf8FJ/0t4Zl9v5bUb/QU5s/bs7Jvdc2QWRLISVnK6j81m9k9ZxBxpaUZ4TUkxuxJzWa1GYL3Z5uBg/NE8/f8ZV/6/Iyttuf2ag0P2N0fHnB13s83fP2ffezGVqUFx7ty5REZGMnnyZEJCQhg3bhxhYWHExsbi4+Nzzf5eXl4MHjyYOnXq4OzszKJFi4iIiMDHx4ewMKP3KjIykuXLl/PVV18REBDATz/9RN++falQoQKdO3fO70ssfqq2BixwdAecOQQlK5hdkRREqalw4YLxl+uUlP/+knvl46ufZ+XxrR5TUALB1eHvZqFOCg+L5b/Nas3Z4+wel53X8uv5jXqa0n9ZzeqW3f2vPMZi+a8XL6tbdve/0TE3CmAuLuDq+l+PY2aPb/Z6dh9f2ZY+BPNWtiv/n5XT4y9dynpQy+nm7Hzj14vjEFwRwGKzmfebUUhICM2aNWPChAkApKWl4e/vT//+/Rk4cGCWztG4cWM6duzIyJEjAahfvz7h4eEMuaJ7u0mTJtxzzz28/fbbWTpnfHw8/v7+HDhwgEqVKmXzqoRP28KhTdDlY2jU0+xq5EZsNuMXlQsX/gtuN3uc1f1udIyG9uS99F+E038hv/IX86y0Xf2L/PX+0ny9vz5npz2758gsfORFW1b3zU6YE7lS+v+Dk5ONQOTkZIQ1Jyd9XkQKkOKaDUzrUUxJSWHjxo0MGjTI3ma1WgkNDWXt2rU3Pd5ms7F8+XJiY2N577337O0tW7bk+++/5/HHH6dChQqsXLmSnTt38sEHH1z3XMnJySQnJ9ufnz17NodXJYCxTMahTcbw0/wKijab8Y/slX+pvfKei8yeZ2Wf7ByTfvP/1a73j/2NfgnI7jHZaU9N/S+0Xbxofi+axfLfX3Ov/KtubjzO7nEF6a/G1wt6WQl3Dg4F61pEJHMWy389eCIiBYxpQfHYsWOkpqbi6+ubod3X15cdO3Zc97jTp09TsWJFkpOTcXBw4OOPP6Z9+/b218ePH8/TTz9NpUqVcHR0xGq1MmXKFFq3bn3dc0ZFRfHWW2/d+kWJoXo7+G0M7FlphKes/sKalgYnT8Lx48Z27Nh/j69+fuwYnDgB58//F9jMDjyFncUCbm7G5uqaP49dXIxgIyIiIiIFSqH7Dc3T05OYmBjOnTtHdHQ0kZGRVKtWjTZt2gBGUPz999/5/vvvqVKlCr/++iv9+vWjQoUKhIaGZnrOQYMGERkZaX9+8OBBAgMD8+NyiqZKzcHqDomJ8Nt3YCl7bcjL7PHJk9fvlcsuJ6eMs8lded/F9dpudR8Hh2vruFF4vd5red1+ZSC8MrRpqJOIiIiI/Mu0oOjt7Y2DgwOJiYkZ2hMTE/Hz87vucVarlRo1agAQHBzM9u3biYqKok2bNly4cIE33niDBQsW0LFjRwAaNmxITEwM//vf/64bFF1cXHC5YtjHmTNnbvXyiqa9e+Gff7LW45f+PRzTLfvvU7IklC0L3t7G16sfpz/38gIPj8zDm4bdiYiIiIjkmGlB0dnZmSZNmhAdHU3Xrl0BYzKb6Ohonn/++SyfJy0tzX5/4aVLl7h06RLWq0KCg4MDabnVU1Wcvf8+TJqUvWNKOEOFgBsHvysfe3kZ94yJiIiIiIhpTB16GhkZSe/evWnatCnNmzdn3LhxJCUlERERAUCvXr2oWLEiUVFRgHEvYdOmTalevTrJycksXryYGTNmMOnf8FKyZEnuvPNOXn31Vdzc3KhSpQq//PILX375JWPHjjXtOouMatWgXr2shb604zArDJzd4PU/wcnV7OpFRERERCSLTA2K4eHhHD16lKFDh5KQkEBwcDBLliyxT3ATFxeXoXcwKSmJvn37Eh8fj5ubG3Xq1OGrr74iPDzcvs+cOXMYNGgQPXv25MSJE1SpUoV33nmHZ599Nt+vr8h55RVjywqbDUpVgLOHIW6tMROqiIiIiIgUCqauo1hQFde1UnLdgufgz1nQ8gW4e6TZ1YiIiIiIZFtxzQaa8UPyTnov4p4V5tYhIiIiIiLZoqAoeadaG+Nrwl9w7qippYiIiIiISNYpKEreKeEDvg2Mx3t/MbcWERERERHJMgVFyVvV2xhf/9HwUxERERGRwkJBUfJWtX/vU/xnuTETqoiIiIiIFHgKipK3qrQEBxc4ewiO7TS7GhERERERyQIFRclbTm5QpYXxWMNPRUREREQKBQVFyXvVtEyGiIiIiEhhoqAoeS99PcW9v8HlFHNrERERERGRm1JQlLzn2wDcveFSEsSvN7saERERERG5CQVFyXtWK1RrYzzW8FMRERERkQJPQVHyR/rwU01oIyIiIiLFwMSJEwkICMDV1ZWQkBDWrVt3w/3HjRtH7dq1cXNzw9/fn5dffpmLFy/mU7XXUlCU/JE+oc2hTXDhpLm1iIiIiIjkoblz5xIZGcmwYcPYtGkTQUFBhIWFceTIkUz3nzVrFgMHDmTYsGFs376dzz//nLlz5/LGG2/kc+X/UVCU/FGqInjXBlsa7P3V7GpERERERPLM2LFjeeqpp4iIiCAwMJDJkyfj7u7O1KlTM91/zZo1tGrVikceeYSAgADuvvtuevTocdNeyLykoCj5R8NPRURERKSQOnv2LGfOnLFvycnJme6XkpLCxo0bCQ0NtbdZrVZCQ0NZu3Ztpse0bNmSjRs32oPhnj17WLx4Mffee2/uX0gWKShK/tF6iiIiIiJSSAUGBlKqVCn7FhUVlel+x44dIzU1FV9f3wztvr6+JCQkZHrMI488wogRI7j99ttxcnKievXqtGnTRkNPpZgIaAVWRzi5D07sMbsaEREREZEs27ZtG6dPn7ZvgwYNyrVzr1y5klGjRvHxxx+zadMm5s+fzw8//MDIkSNz7T2yy9G0d5bix8UT/ENg/2pj+KlXNbMrEhERERHJEk9PT0qWLHnT/by9vXFwcCAxMTFDe2JiIn5+fpkeM2TIEB577DGefPJJABo0aEBSUhJPP/00gwcPxmrN//499ShK/tLwUxEREREpwpydnWnSpAnR0dH2trS0NKKjo2nRokWmx5w/f/6aMOjg4ACAzWbLu2JvQEFR8lf6hDZ7f4XUy+bWIiIiIiKSByIjI5kyZQrTp09n+/btPPfccyQlJREREQFAr169Mgxd7dSpE5MmTWLOnDns3buXZcuWMWTIEDp16mQPjPlNQ08lf1VoBK6l4OJpOLQZ/JuZXZGIiIiISK4KDw/n6NGjDB06lISEBIKDg1myZIl9gpu4uLgMPYhvvvkmFouFN998k4MHD1KuXDk6derEO++8Y9YlYLGZ1ZdZgMXHx+Pv78+BAweoVKmS2eUUPXMfg+3fQ9vBcOdrZlcjIiIiInJdxTUbaOip5D+tpygiIiIiUqApKEr+S5/QJn4dJJ81txYREREREbmGgqLkP6+qUCYA0i7DvlVmVyMiIiIiIldRUBRzVG9nfNXwUxERERGRAkdBUcyh9RRFRERERAosBUUxR9XWYLHCsZ1wOt7sakRERERE5AoKimIOt9JQobHxWMNPRUREREQKFAVFMU/6fYoafioiIiIiUqAoKIp50tdT3LMS0tJMLUVERERERP6joCjmqdQMnEvA+eOQ+JfZ1YiIiIiIyL8UFMU8Dk4QcLvx+J/l5tYiIiIiIiJ2CopiLq2nKCIiIiJS4CgoirnS11OM+x0uXTC3FhERERERARQUxWzeNaFkRUhNhv1rzK5GRERERERQUBSzWSz/9SrqPkURERERkQJBQVHMd+UyGSIiIiIiYjoFRTFftTbG18S/4dwRU0sREREREZECEhQnTpxIQEAArq6uhISEsG7duuvuO3/+fJo2bUrp0qXx8PAgODiYGTNmZNjHYrFkur3//vt5fSmSEx7e4NfQeKxeRRERERER05keFOfOnUtkZCTDhg1j06ZNBAUFERYWxpEjmfcseXl5MXjwYNauXcuWLVuIiIggIiKCpUuX2vc5fPhwhm3q1KlYLBYeeOCB/Losya7quk9RRERERKSgsNhsNpuZBYSEhNCsWTMmTJgAQFpaGv7+/vTv35+BAwdm6RyNGzemY8eOjBw5MtPXu3btytmzZ4mOjs7S+eLj4/H39+fAgQNUqlQpaxcit2bPSviyC5TwgwE7jEluRERERERMVlyzgak9iikpKWzcuJHQ0FB7m9VqJTQ0lLVr1970eJvNRnR0NLGxsbRu3TrTfRITE/nhhx944oknrnue5ORkzpw5Y9/Onj2b/YuRW+N/Gzi6wrkEOLrD7GpERERERIo1U4PisWPHSE1NxdfXN0O7r68vCQkJ1z3u9OnTlChRAmdnZzp27Mj48eNp3759pvtOnz4dT09PunXrdt3zRUVFUapUKfsWGBiYswuSnHNyhSotjcf/rDC3FhERERGRYs70exRzwtPTk5iYGNavX88777xDZGQkK1euzHTfqVOn0rNnT1xdXa97vkGDBnH69Gn7tm3btjyqXG5I6ymKiIiIiBQIjma+ube3Nw4ODiQmJmZoT0xMxM/P77rHWa1WatSoAUBwcDDbt28nKiqKNm3aZNjvt99+IzY2lrlz596wDhcXF1xcXOzPz5w5k80ryRs2m40Ll1LNLiPfWKrciRtg27+aC+eTwNHlpseIiIiISNHl5uSARXNXmMLUoOjs7EyTJk2Ijo6ma9eugDGZTXR0NM8//3yWz5OWlkZycvI17Z9//jlNmjQhKCgot0rOVxcupRI4dOnNdywybKx3KUW5S6d5/O2P+T1NQ4BFREREirNtI8JwdzY1shRbpg89jYyMZMqUKUyfPp3t27fz3HPPkZSUREREBAC9evVi0KBB9v2joqJYtmwZe/bsYfv27YwZM4YZM2bw6KOPZjjvmTNn+Oabb3jyySfz9XrkVlhYlVYfgNutf5lci4iIiIhI8WV6PA8PD+fo0aMMHTqUhIQEgoODWbJkiX2Cm7i4OKzW//JsUlISffv2JT4+Hjc3N+rUqcNXX31FeHh4hvPOmTMHm81Gjx498vV6cpObkwPbRoSZXUa+cthyAhat5jn//TweUbyuXUREREQycnNyMLuEYsv0dRQLouK6VkqBcOYwjK0DWOC1PeDuZXZFIiIiIlKMFddsYPrQU5EMSpaHcnUBG+z9xexqRERERESKJQVFKXiqpy+TofUURURERETMoKAoBU+1K4KiRkaLiIiIiOQ7BUUpeAJagdUJTsfBiT1mVyMiIiIiUuwoKErB4+wBlW8zHv+z3NxaRERERESKIQVFKZiqtTG+7llpZhUiIiIiIsWSgqIUTOkT2uz9FVIvm1uLiIiIiEgxo6AoBVP5YHAtDcln4OBGs6sRERERESlWFBSlYLI6XDH8VMtkiIiIiIjkJwVFKbi0nqKIiIiIiCkUFKXgSl9PMX49XDxtbi0iIiIiIsWIgqIUXGWqgFc1sKXCvlVmVyMiIiIiUmwoKErBVr2d8VXDT0VERERE8o2CohRs6cNPNaGNiIiIiEi+UVCUgq3qHWBxgOO74VSc2dWIiIiIiBQLCopSsLmWgopNjMcafioiIiIiki8UFKXgS79PUcNPRURERETyhYKiFHzp6ynuWQlpqaaWIiIiIiJSHCgoSsFXsQk4e8KFk3D4T7OrEREREREp8hQUpeBzcDImtQENPxURERERyQcKilI4aD1FEREREZF8o6AohUP6eooH/oCUJHNrEREREREp4hQUpXAoWx1K+UNqCuxfY3Y1IiIiIiJFmoKiFA4WC1RrYzzW8FMRERERkTyloCiFh9ZTFBERERHJFwqKUnhUawNY4Mg2OJtgdjUiIiIiIkWWgqIUHu5eUD7IeKzhpyIiIiIieUZBUQqX6v/OfqrhpyIiIiIieUZBUQoX+32KK8FmM7UUEREREZGiSkFRChf/EHByh3OJxr2KIiIiIiKS6xQUpXBxdIEqLY3H/yw3txYRERERkSJKQVEKn2r/3qeoCW1ERERERPKEgqIUPun3Ke5fA5cumluLiIiIiEgRpKAohY9PXSjhB5cvwIE/zK5GRERERKTIUVCUwsdigWptjMe6T1FEREREJNcpKErhVCPU+LphKhyNNbcWEREREZEiRkFRCqfALlC5JSSfgdnd4fwJsysSERERESkyFBSlcHJ0hvAZUKoynNgD8yIg9bLZVYmIiIiIFAkKilJ4eXhDj1ng5AF7VsJPg82uSEREREQEgIkTJxIQEICrqyshISGsW7fuhvufOnWKfv36Ub58eVxcXKhVqxaLFy/Op2qvZXpQzM43cP78+TRt2pTSpUvj4eFBcHAwM2bMuGa/7du307lzZ0qVKoWHhwfNmjUjLi4uLy9DzOLXALp9Yjz+YzJsnG5uPSIiIiJS7M2dO5fIyEiGDRvGpk2bCAoKIiwsjCNHjmS6f0pKCu3bt2ffvn3MmzeP2NhYpkyZQsWKFfO58v+YGhSz+w308vJi8ODBrF27li1bthAREUFERARLly617/PPP/9w++23U6dOHVauXMmWLVsYMmQIrq6u+XVZkt/qdoK2//Ym/jDAWF9RRERERMQkY8eO5amnniIiIoLAwEAmT56Mu7s7U6dOzXT/qVOncuLECRYuXEirVq0ICAjgzjvvJCgoKJ8r/4/FZrPZzHrzkJAQmjVrxoQJEwBIS0vD39+f/v37M3DgwCydo3HjxnTs2JGRI0cC0L17d5ycnDLtacyq+Ph4/P39OXDgAJUqVcrxeSQf2WzGfYpbF4B7WXhqBZSpYnZVIiIiIlLIpWeDbdu2Zejhc3FxwcXF5Zr9U1JScHd3Z968eXTt2tXe3rt3b06dOsV33313zTH33nsvXl5euLu7891331GuXDkeeeQRXn/9dRwcHPLkum7GtB7FlJQUNm7cSGho6H/FWK2Ehoaydu3amx5vs9mIjo4mNjaW1q1bA0bQ/OGHH6hVqxZhYWH4+PgQEhLCwoULb3iu5ORkzpw5Y9/Onj17S9cmJrBYoMvH4NcQzh+HOY9A8jmzqxIRERGRIiIwMJBSpUrZt6ioqEz3O3bsGKmpqfj6+mZo9/X1JSEhIdNj9uzZw7x580hNTWXx4sUMGTKEMWPG8Pbbb+f6dWSVaUExJ99AgNOnT1OiRAmcnZ3p2LEj48ePp3379gAcOXKEc+fO8e6779KhQwd++ukn7r//frp168Yvv/xy3XNGRUVl+KEHBgbmzkVK/nJ2hx6zwcMHEv+GBc9AWprZVYmIiIhIEbBt2zZOnz5t3wYNGpRr505LS8PHx4dPP/2UJk2aEB4ezuDBg5k8eXKuvUd2mT6ZTXZ5enoSExPD+vXreeedd4iMjGTlypWA8Q0G6NKlCy+//DLBwcEMHDiQ++6774bf5EGDBmX4oW/bti0/LkXyQqlK0H0mODjDjkWwMvO/9IiIiIiIZIenpyclS5a0b5kNOwXw9vbGwcGBxMTEDO2JiYn4+fllekz58uWpVatWhmGmdevWJSEhgZSUlNy7iGwwLSjm5BsIxvDUGjVqEBwczIABA3jwwQft3b7e3t44Ojpe0yNYt27dG8566uLikuGH7unpeQtXJqbzbw6dPjQe/zoa/p5vbj0iIiIiUmw4OzvTpEkToqOj7W1paWlER0fTokWLTI9p1aoVu3fvtnd8AezcuZPy5cvj7Oyc5zVnxrSgmJNvYGbS0tJITk62n7NZs2bExsZm2Gfnzp1UqaKJTYqV4EegxfPG44V94VCMqeWIiIiISPERGRnJlClTmD59Otu3b+e5554jKSmJiIgIAHr16pVh6Opzzz3HiRMnePHFF9m5cyc//PADo0aNol+/fmZdAo6mvTPGN7B37940bdqU5s2bM27cuGu+gRUrVrT3GEZFRdG0aVOqV69OcnIyixcvZsaMGUyaNMl+zldffZXw8HBat25N27ZtWbJkCf/3f/9nH54qxUj7EXA0FnYvMya3eWoFePre/DgRERERkVsQHh7O0aNHGTp0KAkJCQQHB7NkyRL7/CxxcXFYrf/12fn7+7N06VJefvllGjZsSMWKFXnxxRd5/fXXzboEc5fHAJgwYQLvv/++/Rv40UcfERISAkCbNm0ICAhg2rRpALz55pvMnTuX+Ph43NzcqFOnDi+++CLh4eEZzjl16lSioqKIj4+ndu3avPXWW3Tp0iXLNWl5jCLk4mmYchcc3wWVmkOfReCY+XhyEREREZGrFddsYHpQLIiK64ehyDr+D0xpa4TGoEeg68fGchoiIiIiIjdRXLNBoZv1VCTbylaHh6aBxQH+nAVrJ5hdkYiIiIhIgaagKMVD9XYQNsp4vGwo7Fpmbj0iIiIiIgWYgqIUHyHPQONeYEuDeY/D0Z1mVyQiIiIicksCAgIYMWLEDZcDzAkFRSk+LBa4dwxUbgHJZ2B2d7hw0uyqRERERERy7KWXXmL+/PlUq1aN9u3bM2fOHPvygbdCQVGKF0dneHgGlPKHE//AN30g9bLZVYmIiIiI5MhLL71ETEwM69ato27duvTv35/y5cvz/PPPs2nTphyfV0FRip8S5aDHbHDygD0r4afBZlckIiIiInJLGjduzEcffcShQ4cYNmwYn332Gc2aNSM4OJipU6eS3cUuFBSlePJrAN0+MR7/MRk2Tje3HhERERGRW3Dp0iW+/vprOnfuzIABA2jatCmfffYZDzzwAG+88QY9e/bM1vkc86hOkYKvbidoOxhWvAM/DADvmlClpdlViYiIiIhk2aZNm/jiiy+YPXs2VquVXr168cEHH1CnTh37Pvfffz/NmjXL1nnVoyjFW+tXIbArpF2CuY/BqdydLUpEREREJC81a9aMXbt2MWnSJA4ePMj//ve/DCERoGrVqnTv3j1b51WPohRvFgt0nQQn9kDCFpjdAx5fCi4lzK5MREREROSm9uzZQ5UqVW64j4eHB1988UW2zqseRRFnd2NyGw8fSPwbFjwDaWlmVyUiIiIiclNHjhzhjz/+uKb9jz/+YMOGDTk+r4KiCECpStB9Jjg4w45FsDLK7IpERERERG6qX79+HDhw4Jr2gwcP0q9fvxyfV0FRJJ1/c+j0ofH419Hw93xz6xERERERuYlt27bRuHHja9obNWrEtm3bcnxeBUWRKwU/Ai2eNx4v7AuHYkwtR0RERETkRlxcXEhMTLym/fDhwzg65nxKGgVFkau1HwE1QuHyBZjzCJy99j88EREREZGC4O6772bQoEGcPn3a3nbq1CneeOMN2rdvn+PzKiiKXM3qAA98DmVrwpmDMPdRuJxsdlUiIiIiItf43//+x4EDB6hSpQpt27albdu2VK1alYSEBMaMGZPj8yooimTGrTQ8MhdcS0H8Ovi/l8BmM7sqEREREZEMKlasyJYtWxg9ejSBgYE0adKEDz/8kL/++gt/f/8cn1frKIpcT9nq8NA0+OpB+HMW+AZCy/5mVyUiIiIikoGHhwdPP/10rp5TQVHkRqq3g7BRsOR1WDYUytWBmjkf6y0iIiIikhe2bdtGXFwcKSkpGdo7d+6co/PlKCgeOHAAi8VCpUqVAFi3bh2zZs0iMDAw15OsiOlCnoEjW2HTlzDvcXgyGsrVMrsqERERERH27NnD/fffz19//YXFYsH27+1SFosFgNTU1BydN0f3KD7yyCOsWLECgISEBNq3b8+6desYPHgwI0aMyFEhIgWWxQL3joHKLSD5DMzuDhdOml2ViIiIiAgvvvgiVatW5ciRI7i7u7N161Z+/fVXmjZtysqVK3N83hwFxb///pvmzZsD8PXXX1O/fn3WrFnDzJkzmTZtWo6LESmwHJ3h4RlQyh9O/APf9IHUy2ZXJSIiIiLF3Nq1axkxYgTe3t5YrVasViu33347UVFRvPDCCzk+b46C4qVLl3BxcQHg559/to97rVOnDocPH85xMSIFWoly0GM2OHnAnpXw02CzKxIRERGRYi41NRVPT08AvL29OXToEABVqlQhNjY2x+fNUVCsV68ekydP5rfffmPZsmV06NABgEOHDlG2bNkcFyNS4Pk1gG6fGI//mAwbp5tbj4iIiIgUa/Xr1+fPP/8EICQkhNGjR7N69WpGjBhBtWrVcnzeHAXF9957j08++YQ2bdrQo0cPgoKCAPj+++/tQ1JFiqy6naDtv72JPwyA/WvNrUdEREREiq0333yTtLQ0AEaMGMHevXu54447WLx4MR999FGOz2ux2XK2inhqaipnzpyhTJky9rZ9+/bh7u6Oj49PjgsqCOLj4/H39+fAgQP2mV1FMrDZjPsUty0Ed294egWUrmx2VSIiIiKSywpjNjhx4gRlypSxz3yaEznqUbxw4QLJycn2kLh//37GjRtHbGxsoQ+JIllisUDXSeDXEM4fg9k9IPmc2VWJiIiISDFy6dIlHB0d+fvvvzO0e3l53VJIhByuo9ilSxe6devGs88+y6lTpwgJCcHJyYljx44xduxYnnvuuVsqSqRQcHY3Jrf5tC0k/g2fhUKFYChVyZgdtbQ/lKoMpSqCk5vZ1YqIiIhIEePk5ETlypVzvFbijeQoKG7atIkPPvgAgHnz5uHr68vmzZv59ttvGTp0qIKiFB+lKkH3mTDtPji63dgy41HuivDof9XjSuBWxuilFBERERHJhsGDB/PGG28wY8YMvLy8cu28OQqK58+ft0/B+tNPP9GtWzesViu33XYb+/fvz7XiRAoF/+bQfyMc+ANOxcHpA3A6Hk4dMB6nnIOko8Z2aFPm53AucUV4TO+RrPxfkPT0A6tD/l6XiIiIiBR4EyZMYPfu3VSoUIEqVarg4eGR4fVNm67z++dN5Cgo1qhRg4ULF3L//fezdOlSXn75ZQCOHDlCyZIlc1SISKFW+t+QdzWbDS6cNILj6QP/hccrHycdNcLkjXokrY5QsuK/4bHStb2TpSqBk2veXqOIiIiIFDhdu3bNk/PmKCgOHTqURx55hJdffpl27drRokULwOhdbNSoUa4WKFKoWSzg7mVs5Rtmvs+lC3D6IJyOuyJIpvdIxsGZQ5B2GU7tN7br8fC5fo9kaX9wLa3hrSIiIiJFzLBhw/LkvDleHiMhIYHDhw8TFBSE1WpMnrpu3TpKlixJnTp1crXI/FYYp8CVIiwtFc4mXL9H8tQBuJR08/M4exqB0a8BhA6HkhXyvHQRERGRwq64ZoMcB8V08fHxAEXqm1ZcPwxSSNmHt6aHx/RhrlfcL5l0NOMxHuXgwalQtbU5NYuIiIgUEgU9G1it1hsuhZHTGVFzNPQ0LS2Nt99+mzFjxnDunLF2nKenJwMGDGDw4MH2HkYRyQcZhrcGZb7PpQtGYDy5D34ebizn8WUXaDcEWr0E+m9WREREpFBasGBBhueXLl1i8+bNTJ8+nbfeeivH581RUBw8eDCff/457777Lq1atQJg1apVDB8+nIsXL/LOO+/kuCARyQNObuBd09iqtILFr0DMTIh+Cw6sg/snGUt0iIiIiEih0qVLl2vaHnzwQerVq8fcuXN54okncnTeHA09rVChApMnT6Zz584Z2r/77jv69u3LwYMHc1RMQVHQu5dFbpnNBpu+hMWvQmoylK4C4TOu3yMpIiIiUkwV1mywZ88eGjZsaB8Bml05Gm924sSJTCesqVOnDidOnMhRISKSjywWaNIbnvjJCImn9sNn7Y3wKCIiIiKF2oULF/joo4+oWLFijs+Ro6AYFBTEhAkTrmmfMGECDRteZwkAESl4KgTDM79ArXuMnsXv+8PCfpBy3uzKRERERCQLypQpg5eXl30rU6YMnp6eTJ06lffffz/H583RPYqjR4+mY8eO/Pzzz/Y1FNeuXcuBAwdYvHhxts83ceJE3n//fRISEggKCmL8+PE0b948033nz5/PqFGj2L17N5cuXaJmzZoMGDCAxx57zL5Pnz59mD59eobjwsLCWLJkSbZrEyny3MpA91mw+gNY/jbEfAWH/4SHp0PZ6mZXJyIiIiI38MEHH2SY9dRqtVKuXDlCQkIoUybnc1DkKCjeeeed7Ny5k4kTJ7Jjxw4AunXrxtNPP83bb7/NHXfckeVzzZ07l8jISCZPnkxISAjjxo0jLCyM2NhYfHx8rtnfy8uLwYMHU6dOHZydnVm0aBERERH4+PgQFhZm369Dhw588cUX9ucuLi45uVSR4sFqhTsGQKVmMO9xSPwLPm0DXSdB3fvMrk5ERERErqNPnz55ct5bXkfxSn/++SeNGzfO1lodISEhNGvWzD6UNS0tDX9/f/r378/AgQOzdI7GjRvTsWNHRo4cCRjfrFOnTrFw4cJsXwMU3htWRXLFmUPwTQQc+N143vIFuGsYOOTo70oiIiIihVpBzwZffPEFJUqU4KGHHsrQ/s0333D+/Hl69+6do/OaunhaSkoKGzduJDQ01N5mtVoJDQ1l7dq1Nz3eZrMRHR1NbGwsrVtnXDh85cqV+Pj4ULt2bZ577jmOHz9+3fMkJydz5swZ+3b27NmcX5RIYVeyAvRZBLf1M56v+Qi+7AxnE8ytS0RERESuERUVhbe39zXtPj4+jBo1KsfnNTUoHjt2jNTUVHx9fTO0+/r6kpBw/V9KT58+TYkSJXB2dqZjx46MHz+e9u3b21/v0KEDX375JdHR0bz33nv88ssv3HPPPdft6YyKiqJUqVL2LTAwMHcuUKSwcnCCDqPgoeng7An7V8MnrWHfarMrExEREZErxMXFUbVq1Wvaq1SpQlxcXI7Pa2pQzClPT09iYmJYv34977zzDpGRkaxcudL+evfu3encuTMNGjSga9euLFq0iPXr12fY50qDBg3i9OnT9m3btm35cyEiBV29rvD0SvAJhHOJML0TrP7QWIdRREREREzn4+PDli1brmn/888/KVu2bI7Pm62bjrp163bD10+dOpWtN/f29sbBwYHExMQM7YmJifj5+V33OKvVSo0aNQAIDg5m+/btREVF0aZNm0z3r1atGt7e3uzevZu77rrrmtddXFwyTHZz5syZbF2HSJHmXQOe/BkWvQxb5sKyoXBgHXT9GFxLmV2diIiISLHWo0cPXnjhBTw9Pe234/3yyy+8+OKLdO/ePcfnzVaP4pXDMzPbqlSpQq9evbJ8PmdnZ5o0aUJ0dLS9LS0tjejoaPuyG1mRlpZGcnLydV+Pj4/n+PHjlC9fPsvnFJErOHvA/Z9Ax7Hg4Aw7Fhmzoib8ZXZlIiIiIsXayJEjCQkJ4a677sLNzQ03Nzfuvvtu2rVrd0v3KObqrKc5MXfuXHr37s0nn3xC8+bNGTduHF9//TU7duzA19eXXr16UbFiRaKiogDjfsKmTZtSvXp1kpOTWbx4MQMHDmTSpEk8+eSTnDt3jrfeeosHHngAPz8//vnnH1577TXOnj3LX3/9laVlMgr6zEYipjq4Eb7uA6fjwNHVCI+NeppdlYiIiEieKCzZYNeuXcTExODm5kaDBg2oUqXKLZ3P9Pnuw8PDOXr0KEOHDiUhIYHg4GCWLFlin+AmLi4Oq/W/js+kpCT69u1LfHw8bm5u1KlTh6+++orw8HAAHBwc2LJlC9OnT+fUqVNUqFCBu+++m5EjR2otRZHcULEJPPMLzH8adi+D7/oaS2nc8z44uZpdnYiIiEixVLNmTWrWrJlr5zO9R7EgKix/NRAxVVoa/PY/WDEKsIFfQ3j4S/C6dtYtERERkcKqoGeDBx54gObNm/P6669naB89ejTr16/nm2++ydF5C+WspyJSAFitcOdr8Nh8cPOChC3w6Z0Qu8TsykRERESKjV9//ZV77733mvZ77rmHX3/9NcfnVVAUkVtTvR08+xtUagYXT8PscPj5LUi9bHZlIiIiIkXeuXPncHZ2vqbdycnpllZzUFAUkVtXqhL0WQzNnzGerxoLM7rCuSOmliUiIiJS1DVo0IC5c+de0z5nzhwCAwNzfF7TJ7MRkSLC0RnuHQ3+zeH7F2Dfb/BJa3hoGlS+zezqRERERIqkIUOG0K1bN/755x/atWsHQHR0NLNmzWLevHk5Pq96FEUkdzV4EJ5eAd614exhmNYR1k4EzZslIiIikus6derEwoUL2b17N3379mXAgAEcPHiQ5cuXU6NGjRyfV0FRRHJfudrw1HKo/wCkXYalb8A3veFizsfJi4iIiEjmOnbsyOrVq0lKSmLPnj08/PDDvPLKKwQFBeX4nAqKIpI3XErAA58b6ytanWDbdzClLSRuM7syERERkSLn119/pXfv3lSoUIExY8bQrl07fv/99xyfT/coikjesVgg5Gmo0MjoUTy+Gz67C+4bB0HhZlcnIiIiUqglJCQwbdo0Pv/8c86cOcPDDz9McnIyCxcuvKWJbEA9iiKSH/ybwTO/QrW2cOk8LHgaFr0Ml5PNrkxERESkUOrUqRO1a9dmy5YtjBs3jkOHDjF+/PhcO7+CoojkDw9vePRbuPN1wAIbpsLUMDhzyOzKRERERAqdH3/8kSeeeIK33nqLjh074uDgkKvnV1AUkfxjdYC2b0DPeeBWBg5tNsLiib1mVyYiIiJSqKxatYqzZ8/SpEkTQkJCmDBhAseOHcu18ysoikj+qxkKT/8CXtXgVBxM7QBHdphdlYiIiEihcdtttzFlyhQOHz7MM888w5w5c6hQoQJpaWksW7aMs2fP3tL5FRRFxBxlqkDEEvAJhHMJ8MU9cCjG7KpEREREChUPDw8ef/xxVq1axV9//cWAAQN499138fHxoXPnzjk+r4KiiJjH0xf6/AAVGsOFEzC9E8TlfBpnERERkeKsdu3ajB49mvj4eGbPnn1L51JQFBFzuXtBr++gSitIPgMz7od/lptdlYiIiEih5eDgQNeuXfn+++9zfA4FRRExn2tJY4KbGqHG8hmzwmH7IrOrEhEREcmxiRMnEhAQgKurKyEhIaxbty5Lx82ZMweLxULXrl3ztsCbUFAUkYLB2R26z4a6nSE1Bb7uBVu+NrsqERERkWybO3cukZGRDBs2jE2bNhEUFERYWBhHjhy54XH79u3jlVde4Y477sinSq9PQVFECg5HZ3jwCwh6BGypMP9pY71FERERkUJk7NixPPXUU0RERBAYGMjkyZNxd3dn6tTr/16TmppKz549eeutt6hWrVo+Vps5BUURKVgcHKHLRGj+NGCDRS/D6g/NrkpERESKubNnz3LmzBn7lpycnOl+KSkpbNy4kdDQUHub1WolNDSUtWvXXvf8I0aMwMfHhyeeeCLXa88JBUURKXisVrhnNNweaTxfNhSWvwM2m7l1iYiISLEVGBhIqVKl7FtUVFSm+x07dozU1FR8fX0ztPv6+pKQkJDpMatWreLzzz9nypQpuV53TjmaXYCISKYsFggdBi4lIHoE/Doaks9ChyjjNREREZF8tG3bNipWrGh/7uLikivnPXv2LI899hhTpkzB29s7V86ZGxQURaRgu2MAuJSExa/AH5Mg5Rx0+hCsDmZXJiIiIsWIp6cnJUuWvOl+3t7eODg4kJiYmKE9MTERPz+/a/b/559/2LdvH506dbK3paWlAeDo6EhsbCzVq1e/xeqzT0NPRaTga/4UdJ0EFitsngHfPgmXU8yuSkREROQazs7ONGnShOjoaHtbWloa0dHRtGjR4pr969Spw19//UVMTIx969y5M23btiUmJgZ/f//8LN9OPYoiUjgEPwJO7kZI3DofUpLg4eng5GZ2ZSIiIiIZREZG0rt3b5o2bUrz5s0ZN24cSUlJREREANCrVy8qVqxIVFQUrq6u1K9fP8PxpUuXBrimPT8pKIpI4VGvKzh7wNxHYddSmPkQ9Jhj3McoIiIiUkCEh4dz9OhRhg4dSkJCAsHBwSxZssQ+wU1cXBxWa8Ee3Gmx2TSN4NXi4+Px9/fnwIEDVKpUyexyRORq+1bDrHBIOQuVmkHPb8CtjNlViYiISBFUXLNBwY6xIiKZCWgFvb8zwmH8eph2H5w7YnZVIiIiIkWGgqKIFE4Vm0CfxeDhA4l/wxf3wOl4s6sSERERKRIUFEWk8PINhMeXQCl/OL4bpt4Dx/8xuyoRERGRQk9BUUQKt7LVIeJH8KoOp+OMnsXEbWZXJSIiIlKoKSiKSOFX2t/oWfSpB+cSYdq9cHCT2VWJiIiIFFoKiiJSNJTwgT6LjHsXL5yE6Z1h/xqzqxIREREplBQURaTocPeCXt9BwB3G0hkzusHun82uSkRERKTQUVAUkaLFxdNYV7Hm3XD5AszqDtu+N7sqERERkUJFQVFEih4nNwifCYFdIe0SfNMbYmabXZWIiIhIoaGgKCJFk6MzPDgVgh8FWxosfBbWTTG7KhEREZFCQUFRRIouqwN0Hg8hzxrPF78Cqz4wtyYRERGRQkBBUUSKNqsVOrwLd7xiPP95OESPAJvN1LJERERECjIFRREp+iwWuGsIhL5lPP9tDPz4OqSlmVuXiIiISAFVIILixIkTCQgIwNXVlZCQENatW3fdfefPn0/Tpk0pXbo0Hh4eBAcHM2PGjOvu/+yzz2KxWBg3blweVC4ihcrtL0HHMcbjdZ/A989DWqqpJYmIiIgURKYHxblz5xIZGcmwYcPYtGkTQUFBhIWFceTIkUz39/LyYvDgwaxdu5YtW7YQERFBREQES5cuvWbfBQsW8Pvvv1OhQoW8vgwRKSyaPQn3fwIWK8TMhHmPw+UUs6sSERERKVBMD4pjx47lqaeeIiIigsDAQCZPnoy7uztTp07NdP82bdpw//33U7duXapXr86LL75Iw4YNWbVqVYb9Dh48SP/+/Zk5cyZOTk75cSkiUlgEdYeHpoPVCbYthDmPwKULZldlSEuFSxd1D6WIiIiYytHMN09JSWHjxo0MGjTI3ma1WgkNDWXt2rU3Pd5ms7F8+XJiY2N577337O1paWk89thjvPrqq9SrV++m50lOTiY5Odn+/OzZs9m8EhEpdAI7wyNzYM6jsHsZfPUg3Pmase5i6iVITfn3a/rjlKvaU67aN+Wqx5ev057ZsVe8bvv3vskyAdD0cWj0GLh7mfqtEhERkeLH1KB47NgxUlNT8fX1zdDu6+vLjh07rnvc6dOnqVixIsnJyTg4OPDxxx/Tvn17++vvvfcejo6OvPDCC1mqIyoqirfeeitnFyEihVeNUHhsPsx8GPavgi9X3fyY/HJyHywbCsvfgfoPGENmKzUxuyoREREpJkwNijnl6elJTEwM586dIzo6msjISKpVq0abNm3YuHEjH374IZs2bcJisWTpfIMGDSIyMtL+/ODBgwQGBuZV+SJSkFRpCb2/h6VvwIWT4OAEDs7/bk7G8NT0x1e2279m1u4MVsfM2zMck8mx1n+HyscuhvVT4PCf8OcsY6vQyAiM9bqBs7u53zcREREp0iw2m3k3wqSkpODu7s68efPo2rWrvb13796cOnWK7777LkvnefLJJzlw4ABLly5l3LhxREZGYrX+d/tlamoqVqsVf39/9u3bd9PzxcfH4+/vz4EDB6hUqVJ2L0tEJHfYbHBwI6z/DP6eD6n/DpF3LQ2NHjWGppatbmqJIiIiRV1xzQamTmbj7OxMkyZNiI6OtrelpaURHR1NixYtsnyetLQ0+z2Gjz32GFu2bCEmJsa+VahQgVdffTXTmVFFRAosiwUqNYX7J0PkdmMdyNKV4eIpWDsBxjeGGd1gx2It8yEiIiK5yvShp5GRkfTu3ZumTZvSvHlzxo0bR1JSEhEREQD06tWLihUrEhUVBRj3EzZt2pTq1auTnJzM4sWLmTFjBpMmTQKgbNmylC1bNsN7ODk54efnR+3atfP34kREcotHWWMdyJb9YffPRi/jrmXwT7SxlfKHphHQqBeUKGd2tSIiIlLImR4Uw8PDOXr0KEOHDiUhIYHg4GCWLFlin+AmLi4uwzDSpKQk+vbtS3x8PG5ubtSpU4evvvqK8PBwsy5BRCT/WB2gVpixndgLG7+ATTPg9AGIHgEroqBeV+NeRv8Qo1dSREREJJtMvUexoCqu45BFpJC6dBG2LjB6GQ9u+K/dtwE0ewIaPAQuJcyrT0REpBArrtnA1HsURUQkFzi5QnAPeCoanl5pTHTj6AqJf8Gil2BsXfjxdTi60+xKRUREpJBQUBQRKUoqNIIuE43Jb8JGgVc1SD4Df0yGic1geifY9h2kXjK7UhERESnATL9HUURE8oC7F7ToByHPwZ4VsP5z2Pkj7P3V2DzLQ5MIaNIbPP3MrlZEREQKGPUoiogUZVYr1LgLesyCF/+EOwaAuzecPQwrR8EH9eDr3rBvlbFuo4iIiAgKiiIixUfpynDXUIjcBg98DpVbQNpl2LYQpnWEj2+DdVPg4hmzKxURERGTKSiKiBQ3ji7Q4EF4fAk8u8oYgurkAUd3wOJXjMlvFkVC4lazKxURERGTKCiKiBRnfg2g0zgYsB3uGQ3etSDlHGz4HCa1hKn3wNaFGpYqIiJSzCgoiogIuJaCkGeg3zro/X9QtzNYHCBuDXzTG6aGwcFNZlcpIiIi+URBUURE/mOxQNXWED4DXv4bWr9qDEs98AdMaQff9YOziWZXKSIiInlMQVFERDJXsgK0exP6b4CG3QEbbP4KxjeB1R/B5RSzKxQREZE8oqAoIiI3VrICdPsEnlgGFRpDyllYNsSYJTV2ie5fFBERKYIUFEVEJGv8m8OT0dDlY/DwgRP/wOxwmPkgHN1pdnUiIiKSixQURUQk66xWaNQT+m+EVi+BgzPs/hkmtYAlg+DCKbMrFBERkVygoCgiItnnWhLavwV9f4fa90LaZfj9YxjfGDZ8AWmpZlcoIiIit0BBUUREcq5sdegxGx79Frxrw/njsOgl+PRO2Lfa7OpEREQkhxQURUTk1tUIhedWQ4f3jDUZE/6CaffCN33g1AGzqxMREZFsUlAUEZHc4eAEtz0L/TdB08fBYoWtC2BCU1gRBSnnza5QREREskhBUUREcpeHN9z3ATzzK1S5HS5fhF/ehQnN4O9vtZyGiIhIIaCgKCIiecOvAfRZBA9Nh1KV4Uw8zHscvrgXDv9pdnUiIiJyAwqKIiKSdywWqNcVnl8HbQeDoxvErYFP7oTvX4BzR82uUERERDKhoCgiInnPyQ3ufA36b4D6DwI22DQdxjeBtRPhcorZFYqIiMgVFBRFRCT/lKoED34OEUugfBAkn4alb8CklrDrZ7OrExERkX8pKIqISP6r0gKeWgGdx4NHOTi+C2Y+ADMfhmO7za5ORESk2FNQFBERc1gdoHEv6L8RWjwPVkfYtRQ+vg2WDoaLp82uUEREpNhSUBQREXO5loKwd6Dv71AzDNIuwdoJxv2Lm76EtDSzKxQRESl2FBRFRKRg8K4JPb+GnvOgbE1IOgrf94cpbSHud7OrExERKVYUFEVEpGCp2R6eWwNho8ClJByOgalhMO8JOH3Q7OpERESKBUezCxAREbmGozO06AcNHoblI40hqH/Pg20LoXQVKFMFygT8+zjg360KuJUxt24REZEiQkFRREQKrhLloPNH0OwJ+HEgxK2BE/8YW2ZcS10RHq8IkaUDoLQ/OLrkX+0iIiKFmIKiiIgUfOWDIGIxnDkIJ/f9u+03vp769+u5RGOm1IQtxnYNC5SscEV4vCpQlvAFiyXfLklERKQgU1AUEZHCwWKBUpWMLeD2a19POQ+n4jKGxysD5aUkI2ieOQj7V197vKPrteHxyucunnl4cSIiIgWLgqKIiBQNzu7gU8fYrmazwfnjV4THfRkD5emDcPkiHIs1tsy4l80YHr2qQq0OUMInjy5IRETEPAqKIiJS9Fks4OFtbJWaXvt66iU4HX9Vb+QVvZIXThhB8/xxOLjxv+McXCC4B7ToD9418udaRERE8oGCooiIiIOT0UPoVTXz1y+e+TdAXnFfZPx6OLQZNk6DjdOhTkdo+QJUDsnPykVERPKEgqKIiMjNuJYEvwbGls5mg7i1sPoj2Pkj7FhkbJWaQ6sXoHZHsGq5YhERKZz0L5iIiEhOWCxQpSU8Mgf6rYNGj4GDM8Svg7mPwsRmsOELuHTR7EpFRESyTUFRRETkVpWrDV0mwEt/wx0DjPUcj++GRS/BuPrwy2g4f8LsKkVERLJMQVFERCS3ePrCXUPh5a0QFgWl/CHpKKx4Bz6oB4tfNe5xFBERKeAUFEVERHKbiye06AsvbIYHPge/hnDpPKz7FD5qBN/0gYObzK5SRETkuhQURURE8oqDEzR4EJ75FXp9B9XvAlsabF0AU9rCtPtg50/GxDgiIiIFSIEIihMnTiQgIABXV1dCQkJYt27ddfedP38+TZs2pXTp0nh4eBAcHMyMGTMy7DN8+HDq1KmDh4cHZcqUITQ0lD/++COvL0NERCRzFgtUawOPzYdnV0PD7mB1hH2/wayH4OMWsHkmXE4xu1IRERGgAATFuXPnEhkZybBhw9i0aRNBQUGEhYVx5MiRTPf38vJi8ODBrF27li1bthAREUFERARLly6171OrVi0mTJjAX3/9xapVqwgICODuu+/m6NGj+XVZIiIimfOrD90+gRf/hBbPg7MnHN0O3/WFDxvCqnFw8bTZVYqISDFnsdnMHe8SEhJCs2bNmDBhAgBpaWn4+/vTv39/Bg4cmKVzNG7cmI4dOzJy5MhMXz9z5gylSpXi559/5q677rrm9eTkZJKTk+3PDx48SGBgIAcOHKBSpUo5uCoREZEsunAKNk6DPybD2cNGm7MnNOkNtz0HpfTvUJEWv9G4d3XPCmj9KjR/yuyKROQq8fHx+Pv7ZzsbTJw4kffff5+EhASCgoIYP348zZs3z3TfKVOm8OWXX/L3338D0KRJE0aNGnXd/fODqT2KKSkpbNy4kdDQUHub1WolNDSUtWvX3vR4m81GdHQ0sbGxtG7d+rrv8emnn1KqVCmCgoIy3ScqKopSpUrZt8DAwJxdkIiISHa5lYbbX4IXt0CXj6FcXUg5C2snwIdBMP8ZSPjb7ColN126CDGz4NO28Fk72DIHziXC4ldg3RSzqxORXJDdUZMrV66kR48erFixgrVr1+Lv78/dd9/NwYMH87ny/5jao3jo0CEqVqzImjVraNGihb39tdde45dffrnufYWnT5+mYsWKJCcn4+DgwMcff8zjjz+eYZ9FixbRvXt3zp8/T/ny5Vm4cCHNmjXL9HzqURQRkQLDZoNdy2DNR8Y9jOmq3wWtXoCqdxr3PErhcyoONkyFjdPhwr/rajo4Q71u4OwBGz432u4bB00jTCtTRDLKSY/irY6aTE1NpUyZMkyYMIFevXrdUv055WjKu94iT09PYmJiOHfuHNHR0URGRlKtWjXatGlj36dt27bExMRw7NgxpkyZwsMPP8wff/yBj4/PNedzcXHBxcXF/vzMmTP5cRkiIiLXslig1t3GdnCTERi3fQf/RBubX0No9SIEdgWHQvnPePFis8GelUZP4c4fjVlvAUpWgmaPQ+Pe4OFt7OfkZvQkL3rJmOyo8WNmVi4iVzl79myGnHB1hkiXPmpy0KBB9rbsjJoEOH/+PJcuXcLLy+vWC88hU/+F8fb2xsHBgcTExAztiYmJ+Pn5Xfc4q9VKjRo1AAgODmb79u1ERUVlCIoeHh7UqFGDGjVqcNttt1GzZk0+//zzDD8wERGRAq1iY3hoGpzYC79/DJu/goQt8O0T8PNwuK0vNO4FLiXMrlSudvEM/DnbCIjHd/3XXvVOaP401OqQMehbLHD325CWCn9Mgu/7g9UBgh/J/9pFJFNX3542bNgwhg8ffs1+x44dIzU1FV9f3wztvr6+7NixI0vv9frrr1OhQoUMt+jlN1ODorOzM02aNCE6OpquXbsCRrdsdHQ0zz//fJbPk5aWlmHoaE73ya7U1FQuXbqUq+eUosXJyQkHBwezyxCRws6rKtz7PrQZBOs/gz8+gdMHYOkg+OVdaPoEhDwLnr43P5fkrSPbjXD45xy4lGS0OZcwAl+zJ6Fc7esfa7FAhyhIuwzrp8DCvkbPYsOH86d2Ebmhbdu2UbFiRfvzzHoTc8O7777LnDlzWLlyJa6urnnyHllh+piVyMhIevfuTdOmTWnevDnjxo0jKSmJiAhjbH6vXr2oWLEiUVFRgDHxTNOmTalevTrJycksXryYGTNmMGnSJACSkpJ455136Ny5M+XLl+fYsWNMnDiRgwcP8tBDD+VKzTabjYSEBE6dOpUr55OirXTp0vj5+WHRPUUicqvcveDO16Blf6O3as0EOPEPrBprDFkM6g4t+kO5WmZXWrykXobYH4yAeOV9pd61jVlMg7qDi2fWzmWxwD2jjbC48QtY8IzRs1j/gbypXUSyzNPTk5IlS950v5yOmgT43//+x7vvvsvPP/9Mw4YNb6neW2V6UAwPD+fo0aMMHTqUhIQEgoODWbJkib2rNi4uDqv1v8lZk5KS6Nu3L/Hx8bi5uVGnTh2++uorwsPDAXBwcGDHjh1Mnz6dY8eOUbZsWZo1a8Zvv/1GvXr1cqXm9JDo4+ODu7u7AoBkymazcf78efvsVuXLlze5IhEpMpzcoOnj0LgPxC427mM88Ads+tLYat1jTHxTuYUmvslL544YE9Ns/ALO/DszocUKdTpCs6egauucff+tVug41giLm2fAt0+BxQHqdc3V8gu8yynGH0GsjtDqJd2TK4VGTkdNjh49mnfeeYelS5fStGnTfKr2+kxfR7EgutHMRqmpqezcuRMfHx/Kli1rUoVSmBw/fpwjR45Qq1YtDUMVkbwT9zus/sgIjvz7T3vFpkZgrNYWHJyMX7gtDkYQkZyx2SB+vdF7uHUBpP17C4q7t7H2ZZMIKO2fO++VlgbfPw8xM42f3UPToe59uXPugu5sInz9mPEHEIAa7eHBqeB6894ckdyWk1lP586dS+/evfnkk0/soya//vprduzYga+v7zWjJt977z2GDh3KrFmzaNWqlf08JUqUoEQJc+5D159msin9nkR3d3eTK5HCIv2zcunSJQVFEck7lW8ztmO7jGGoMbPh4Ab4OrNp1S1G8LBvDv9uVz53vCJYXtVmdTTC5jXnuHL/q4/59z1cS0PZGuBdE7yqg5N5999ky6UL8Pe3sO5TOPznf+2Vmhm9h/W6gmMu369ktULn8UbP4pa58E0fCJ8Bte/J3fcpaA5ugjk94ewhcCkFqSmwexlM7QA9v4ZSWrpMCr7sjpqcNGkSKSkpPPjggxnOc70Jc/KDehQzcaO/Gly8eJG9e/dStWpVU28ulcJDnxkRMcW5I0aoWf/5f2v2FTgWo/etbE0jOKYHyLI1oWSFgjFs9uQ+43u4eQZcOGm0ObhAgweNyWkqNs77GlIvw4KnjaDq4AzhM43lU4qiP+fA9y9AajJ414LusyH5NMzqDklHoIQfPDIHKjQyu1IpRnLSo1gUqEdRRESkKCrhA+3ehDZvGD0yaZf/3VKNr7bUa9vsj698/u9mS7uqLfU6x155zFXnSr0E548bvZ7Hd8HF08Yi9KfijDUir+TkAWWr/xcc04Nk2Rp5vxxIWhrsWf7v2odLsQ/lLVUZmj0BjR4Dj3y8/cTBEe7/1PheblsIcx+FHrOhxl35V0NeS70MPw8zesPBuM+226f/DTV9KhpmhcORbfDFvfDAZ8a9oCKSZxQURUREijKrFawFcDSDzQZJx4zAmB4cj+2GYzuNXrxLScaakQlbrj3WswJ417giQNY0npfyN4a35tSFUxAzy1ia4sSe/9qrtzPWPqx5962d/1Y4OBrhKO0y7FgEcx6BR+ZCtTbm1JObzp+AeY/DnhXG89avGn/guPJe2tKV4fElxvDbf5YbQ1Pvfhta9CsYPc8iRZCCouRIQEAAL730Ei+99NItn2vlypW0bduWkydPUrp06Vs+n4iIFAIWC5QoZ2xVWmZ87XKKERavDpHHdxk9kmcPGdveXzMe5+hq3PeYWYh0LXX9WhK3Gr2HW+bCpfNGm0tJCO5pDC/1rpGrl55jDk7w4BfGfac7fzSGY/b8BqreYXZlOXdkO8zuASf3gpM7dJ10/dldXUvBI9/Aj6/Chqnw02BjeZh73teMqCJ5QP9VFSNt2rQhODiYcePG3fK51q9fj4eHx60XJSIicjVHZ2MtyMzWgzx/Ao7vviJA7jKen9gDly/Cka3GdjUPn2vvg0w5ZwSO/av/288n0AiHDcPzfohrTjg6w8PTjeGnu36CWQ/Do99eG7YLg+2LjHUiU84ZPYbdZ4Nf/Rsf4+BoLB3iVR1+etP4+Z3cDw9N04yoIrlMQVHsbDYbqampODre/GNRrly5fKhIRETkKu5e4N4c/JtnbE+9DKfj/ut5TA+Qx3bBuQRjIpSkIxlDYTqLA9TtBM2fgiqtCv5QRkcXeHgGzOlhDMP86kF4bAFUDjG7sqxJS4NfR8NKY1kAAu4wlv7I6n2fFgu0fB7KBMD8p4z7W6d2MIbi5tbSJCKCFlLKBTabjfMpl/N9y86EtX369OGXX37hww8/xGKxYLFYmDZtGhaLhR9//JEmTZrg4uLCqlWr+Oeff+jSpQu+vr6UKFGCZs2a8fPPP2c4X0BAQIaeSYvFwmeffcb999+Pu7s7NWvW5Pvvv8/x9/Tbb7+lXr16uLi4EBAQwJgxYzK8/vHHH1OzZk1cXV3x9fXNMJXwvHnzaNCgAW5ubpQtW5bQ0FCSkpJyXIuIiBQCDo7gVc2YDbRFP+g0DvosgldiYeABeGq5MSFM61chsCv41jf2b/0avPy30UsXcHvBD4npnFyh+yyoeqdxP+dXD0D8BrOrurnks8b6iOkhMeRZI+TmZHKguvdBxGIo4Wv0In92l7G0hojkCvUo5oILl1IJHLo0399324gw3J2z9iP88MMP2blzJ/Xr12fEiBEAbN1qDM0ZOHAg//vf/6hWrRplypThwIED3Hvvvbzzzju4uLjw5Zdf0qlTJ2JjY6lcufJ13+Ott95i9OjRvP/++4wfP56ePXuyf/9+vLy8snVdGzdu5OGHH2b48OGEh4ezZs0a+vbtS9myZenTpw8bNmzghRdeYMaMGbRs2ZITJ07w22+/AXD48GF69OjB6NGjuf/++zl79iy//fZbtkK1iIgUMa4loWITYytKnNygxxxj+Om+32DG/dBrYcG9zhN7YPYjcHS7sczHfR9Ao0dv7ZwVGsGT6TOibv13RtQpRg+xiNwS9SgWE6VKlcLZ2Rl3d3f8/Pzw8/OzL/4+YsQI2rdvT/Xq1fHy8iIoKIhnnnmG+vXrU7NmTUaOHEn16tVv2kPYp08fevToQY0aNRg1ahTnzp1j3bp12a517Nix3HXXXQwZMoRatWrRp08fnn/+ed5//33AWKDUw8OD++67jypVqtCoUSNeeOEFwAiKly9fplu3bgQEBNCgQQP69u1LiRIF8D4TERGRW+Xsbgy5rNwSks8YYfFQjNlVXeuf5fBpWyMklvCDPotvPSSmK+1vzIhaIxQuX4C5j8Hqj4yZdUUkx9SjmAvcnBzYNiLMlPfNDU2bNs3w/Ny5cwwfPpwffvjBHrwuXLhAXFzcDc/TsGFD+2MPDw9KlizJkSNHsl3P9u3b6dKlS4a2Vq1aMW7cOFJTU2nfvj1VqlShWrVqdOjQgQ4dOtiHvAYFBXHXXXfRoEEDwsLCuPvuu3nwwQcpU6ZMtusQEREpFJw9oOfXxvDTA3/AjK7Q+//Ar4HZlRlhbe1EWDbEWIuzYlMI/wpKls/d93EtCT3mwpLXYf1nxvud2AP3vm/MFisi2aYexVxgsVhwd3bM982SS/dRXD176SuvvMKCBQsYNWoUv/32GzExMTRo0ICUlJQbnsfJKeP/iC0WC2lpablS45U8PT3ZtGkTs2fPpnz58gwdOpSgoCBOnTqFg4MDy5Yt48cffyQwMJDx48dTu3Zt9u7dm+t1iIiIFBguntBznhHELpyE6Z2NZT/MdOkCLHjWWMbClgbBj0KfH3I/JKZzcIR7/wdhUYAFNn4BMx+Ci6fz5v1EijgFxWLE2dmZ1NTUm+63evVq+vTpw/3330+DBg3w8/Nj3759eV/gv+rWrcvq1RlnpVu9ejW1atWyD5d1dHQkNDSU0aNHs2XLFvbt28fy5csBI6C2atWKt956i82bN+Ps7MyCBQvyrX4RERFTuJaEx+Yb9+1dOGGExSM7zKnl9EH44h7YMseYVfae0dBlgjEJT16yWKBFX2OiHyd32LMCPg8zltAQkWxRUCxGAgIC+OOPP9i3bx/Hjh27bm9fzZo1mT9/PjExMfz555888sgjedIzeD0DBgwgOjqakSNHsnPnTqZPn86ECRN45ZVXAFi0aBEfffQRMTEx7N+/ny+//JK0tDRq167NH3/8wahRo9iwYQNxcXHMnz+fo0ePUrdu3XyrX0RExDSupYxZRP0awvljML0THN2ZvzXE/QGftoFDm8HNy6gn5Jn8nVG2zr3/zojqZ9wX+dldEL8x/95fpAhQUCxGXnnlFRwcHAgMDKRcuXLXvedw7NixlClThpYtW9KpUyfCwsJo3LhxvtXZuHFjvv76a+bMmUP9+vUZOnQoI0aMoE+fPgCULl2a+fPn065dO+rWrcvkyZOZPXs29erVo2TJkvz666/ce++91KpVizfffJMxY8Zwzz335Fv9IiIipnIrA72+A98GxtqR0zvB8X/y5703ToNpHY339a0PT6+Aanfmz3tfrUIjeCraqCPpKEy7F7Z9Z04tIoWQxaZ1A64RHx+Pv78/Bw4coFKlShleu3jxInv37qVq1aq4uubx8AkpEvSZERERUyQdh+n3wZFt4FkBIn4w1o7MC6mXYMlAYyIZgMAu0OVjcCkAs44nn4V5j8Oun4znoW9BqxcLz5qZYrobZYOiTD2KIiIiIkWRR1no9T2UqwNnD8G0TnByX+6/z7mj8GWXf0OiBdq9CQ9NLxghEYyJfrrPhubPGM9/Hgb/94IRbkXkuhQUJc89++yzlChRItPt2WefNbs8ERGRoqtEOSMslq0JZ+KNYainbrzcVbYc/hOmtIX9q8HZE3rMhtavFrzeOgdHuHc0dHgPLFbY9CXMfBAunDK7MpECS+soSp4bMWKEfSKaq5UsWTKfqxERESlmPH2NdRWndYQT/xhhsc8PUOoWh9D9NQ++e95Y5N6ruhESy9XOnZrzym3PQpkAYyjqnpUwNQwemWu0iUgGCoqS53x8fPDx8TG7DBERkeKrZPl/w+K9xvDT9LBYskL2z5WWCstHwqoPjOc1QuGBz8GtdG5WnHdqd4DHf4RZ4XB0B3wWCj3mQKWmZlcmUqBo6KmIiIhIcVCqIvReBKUrw4k9Rlg8m5C9c1w4ZQSs9JDY6iV45OvCExLTlQ+Cp5aDX4N/Z0TtCFsXml2VSIGioCgiIiJSXJT2N8JiKX84vhumd4ZzR7J27NGdxnqEu5eBo5vRi9j+LbA65G3NeaVkBYhYArU6wOWL8E1v+G0saEEAEUBBUURERKR4KVMFen8PJSvCsVgjLCYdu/ExsUuMkHh8txEyn1gKDR7Mn3rzkksJ6D4LQv6dXC/6Lfj+ebicYm5dIgWAgqKIiIhIceNVzbhn0bM8HN1uLG9x/sS1+9ls8Ov/YHZ3SD4DlVvCUyuMoZtFhdUB7nkP7nnfmBF181cw8wG4cNLsykRMpaAoIiIiUhyVrW6ExRK+kPg3fNk5Y1hMSYJv+hgT12CDpk9Ar++MJTeKopCnjUltnEvA3l/h87vhxF6zqxIxjYKiZFlAQADjxo3L0r4Wi4WFCxfmaT0iIiJyi7xrGussepSDhL9gxv3GhDUn9xlBadtCsDpBpw/hvrHg6GxywXmsVhg8vuTfYbk7jRlRD6wzuyoRUygoioiIiBRnPnWMsOheFg7HwLT74NO2Ri+jhw/0WQRN+phdZf7xawBPRhvDa88fM74ff39rdlUi+U5BUURERKS48w00hpW6lYHEv+DCCajQCJ5eCZVvM7u6/FeyPET8CLXvhdRkmPe4ca+mZkSVYkRBMTfYbMY4/vzesvE/q08//ZQKFSqQlpaWob1Lly48/vjj/PPPP3Tp0gVfX19KlChBs2bN+Pnnn3PtW/TXX3/Rrl073NzcKFu2LE8//TTnzp2zv75y5UqaN2+Oh4cHpUuXplWrVuzfvx+AP//8k7Zt2+Lp6UnJkiVp0qQJGzZsyLXaREREBKMnrdd34FPP6EGM+NFYe7G4cvaA8K/gtn7G8+Uj4bt+mhFVig1HswsoEi6dh1EV8v993zhk/E8sCx566CH69+/PihUruOuuuwA4ceIES5YsYfHixZw7d457772Xd955BxcXF7788ks6depEbGwslStXvqUyk5KSCAsLo0WLFqxfv54jR47w5JNP8vzzzzNt2jQuX75M165deeqpp5g9ezYpKSmsW7cOi8UCQM+ePWnUqBGTJk3CwcGBmJgYnJycbqkmERERyUT5IOi7xuwqCg6rA3QYBWWrweJXIWYmnIqD8BlG76tIEaagWEyUKVOGe+65h1mzZtmD4rx58/D29qZt27ZYrVaCgv6b6nrkyJEsWLCA77//nueff/6W3nvWrFlcvHiRL7/8Eg8PI9hOmDCBTp068d577+Hk5MTp06e57777qF69OgB169a1Hx8XF8err75KnTp1AKhZs+Yt1SMiIiKSLc2ehNJVjFlg9/0Gn7WHnl8by4yIFFEKirnByd3o3TPjfbOhZ8+ePPXUU3z88ce4uLgwc+ZMunfvjtVq5dy5cwwfPpwffviBw4cPc/nyZS5cuEBcXNwtl7l9+3aCgoLsIRGgVatWpKWlERsbS+vWrenTpw9hYWG0b9+e0NBQHn74YcqXLw9AZGQkTz75JDNmzCA0NJSHHnrIHihFRERE8kXN9vD4UpgVDsd3GTOidp9VPO/hlGJB9yjmBovFGAKa39u/QzOzqlOnTthsNn744QcOHDjAb7/9Rs+ePQF45ZVXWLBgAaNGjeK3334jJiaGBg0akJKSP+Pwv/jiC9auXUvLli2ZO3cutWrV4vfffwdg+PDhbN26lY4dO7J8+XICAwNZsGBBvtQlIiIiYudXH56KhvLBcP44TO8Mf80zuyqRPKGgWIy4urrSrVs3Zs6cyezZs6lduzaNGzcGYPXq1fTp04f777+fBg0a4Ofnx759+3LlfevWrcuff/5JUlKSvW316tVYrVZq165tb2vUqBGDBg1izZo11K9fn1mzZtlfq1WrFi+//DI//fQT3bp144svvsiV2kRERESyxdMPIhZDnfuMGVG/fQJ+Ga0ZUaXI0dDTYqZnz57cd999bN26lUcffdTeXrNmTebPn0+nTp2wWCwMGTLkmhlSb+U9hw0bRu/evRk+fDhHjx6lf//+PPbYY/j6+rJ3714+/fRTOnfuTIUKFYiNjWXXrl306tWLCxcu8Oqrr/Lggw9StWpV4uPjWb9+PQ888ECu1CYiIiKSbc4e8PCXsGworJ0AK96BQzHgXRNsqZCW9u/X1Ku+Ztaelsl+V7SnXb7Jvpmc09EFgnrA7S+Baymzv1tSSCkoFjPt2rXDy8uL2NhYHnnkEXv72LFjefzxx2nZsiXe3t68/vrrnDlzJlfe093dnaVLl/Liiy/SrFkz3N3deeCBBxg7dqz99R07djB9+nSOHz9O+fLl6devH8888wyXL1/m+PHj9OrVi8TERLy9venWrRtvvfVWrtQmIiIikiNWBwh7x5jQZvGrEPsDxJpd1BVWjYWN06D1q9DsCSM8imSDxWZTP/nV4uPj8ff358CBA1SqVCnDaxcvXmTv3r1UrVoVV1dXkyqUwkSfGRERkSIu7nfY9p3x2GI1QqTF4aqvt9hudcz6vsd2wvK3ja9gzNjabgjUfwCsuvMsu26UDYoy9SiKiIiIiNyKyrcVrNlPfepC7Y4Q8xWsiIJT+2H+k7B2PIS+BdXbml2hFAL6k4Jk28yZMylRokSmW7169cwuT0REREQcHKFJH3hhE7R7E5w94fCfMKMrzLgfDm8xu0Ip4ApEUJw4cSIBAQG4uroSEhLCunXrrrvv/Pnzadq0KaVLl8bDw4Pg4GBmzJhhf/3SpUu8/vrrNGjQAA8PDypUqECvXr04dMiEdQ6LqM6dOxMTE5PptnjxYrPLExEREZF0zh7GfYovxkDIs2B1gn+Wwyd3wLdPwcn9ZlcoBZTpQ0/nzp1LZGQkkydPJiQkhHHjxhEWFkZsbCw+Pj7X7O/l5cXgwYOpU6cOzs7OLFq0iIiICHx8fAgLC+P8+fNs2rSJIUOGEBQUxMmTJ3nxxRfp3LkzGzZsMOEKix5PT088PT3NLkNEREREssrDG+55zwiLy9+Gv+fBX1/DtoXQ7Clo/Qq4e5ldpRQgpk9mExISQrNmzZgwYQIAaWlp+Pv7079/fwYOHJilczRu3JiOHTsycuTITF9fv349zZs3Z//+/VSuXPmm59NkNpKb9JkRERGRAufQZlg2DPb+Yjx3KWUsp3Hbc+DkZmppBU1xnczG1KGnKSkpbNy4kdDQUHub1WolNDSUtWvX3vR4m81GdHQ0sbGxtG7d+rr7nT59GovFQunSpTN9PTk5mTNnzti3s2fPZvtaREREREQKjQqNoNd38Oi34NsAkk9D9FvwUWPY9KWxHqMUa6YGxWPHjpGamoqvr2+Gdl9fXxISEq573OnTpylRogTOzs507NiR8ePH0759+0z3vXjxIq+//jo9evSgZMmSme4TFRVFqVKl7FtgYGDOL0pEREREpDCwWKBGKDzzK9z/CZTyh7OH4Pv+MKklxP4IWkmv2CoQk9lkl6enJzExMaxfv5533nmHyMhIVq5cec1+ly5d4uGHH8ZmszFp0qTrnm/QoEGcPn3avm3bti0PqxcRERERKUCsVgjqDs9vgLvfAdfScHQHzO4OX9wL8ZrnozgydTIbb29vHBwcSExMzNCemJiIn5/fdY+zWq3UqFEDgODgYLZv305UVBRt2rSx75MeEvfv38/y5cuv25sI4OLigouLi/35mTNncnhFIiIiIiKFlJMrtHweGj0Kqz6APyZD3Br47C6o2xnuGgbeNcyuUvKJqT2Kzs7ONGnShOjoaHtbWloa0dHRtGjRIsvnSUtLIzk52f48PSTu2rWLn3/+mbJly+Zq3cVVQEAA48aNM7sMEREREclLbqWh/VvQfyMEPwoWK2z/HiY2h0WRcO6I2RVKPjB96GlkZCRTpkxh+vTpbN++neeee46kpCQiIiIA6NWrF4MGDbLvHxUVxbJly9izZw/bt29nzJgxzJgxg0cffRQwQuKDDz7Ihg0bmDlzJqmpqSQkJJCQkEBKSoop1ygiIiIiUuiUqgRdJ8Kzq6FWB7ClwobP4cNgWBEFyZoAsigzfR3F8PBwjh49ytChQ0lISCA4OJglS5bYJ7iJi4vDav0vzyYlJdG3b1/i4+Nxc3OjTp06fPXVV4SHhwNw8OBBvv/+e8AYlnqlFStWZBieKsVHamoqFoslw2dJRERERLLANxAemQv7VhlLahzcAL+8a4TGO1+HJn3AwcnsKiWXFYjfmp9//nn2799PcnIyf/zxByEhIfbXVq5cybRp0+zP3377bXbt2sWFCxc4ceIEa9assYdEMIZH2my2TLc8C4k2GyQl5f+WjVmoPv30UypUqEBaWlqG9i5duvD444/zzz//0KVLF3x9fSlRogTNmjXj559/zvG3ZOzYsTRo0AAPDw/8/f3p27cv586dy7DP6tWradOmDe7u7pQpU4awsDBOnjwJGMOJR48eTY0aNXBxcaFy5cq88847gPGZsFgsnDp1yn6umJgYLBYL+/btA2DatGmULl2a77//nsDAQFxcXIiLi2P9+vW0b98eb29vSpUqxZ133smmTZsy1HXq1CmeeeYZfH19cXV1pX79+ixatIikpCRKlizJvHnzMuy/cOFCPDw8tKyKiIiIFG0Bt8OTP8PDX4JXdUg6CotfMYakbl2gGVKLmAIRFAu98+ehRIn8386fz3KJDz30EMePH2fFihX2thMnTrBkyRJ69uzJuXPnuPfee4mOjmbz5s106NCBTp06ERcXl6NvidVq5aOPPmLr1q1Mnz6d5cuX89prr9lfj4mJ4a677iIwMJC1a9eyatUqOnXqRGqqsWbPoEGDePfddxkyZAjbtm1j1qxZ1yyjcjPnz5/nvffe47PPPmPr1q34+Phw9uxZevfuzapVq/j999+pWbMm9957rz3kpaWlcc8997B69Wq++uortm3bxrvvvouDgwMeHh50796dL774IsP7fPHFFzz44IN4enrm6HslIiIiUmhYLBDYBfr9AR3HgIcPnNgD3/QxJr3Zt8rsCiW32OQaBw4csAG2AwcOXPPahQsXbNu2bbNduHDhv8Zz52w2428o+budO5et6+rSpYvt8ccftz//5JNPbBUqVLClpqZmun+9evVs48ePtz+vUqWK7YMPPsjWe6b75ptvbGXLlrU/79Gjh61Vq1aZ7nvmzBmbi4uLbcqUKZm+vmLFChtgO3nypL1t8+bNNsC2d+9em81ms33xxRc2wBYTE3PDulJTU22enp62//u//7PZbDbb0qVLbVar1RYbG5vp/n/88YfNwcHBdujQIZvNZrMlJibaHB0dbStXrrzue2T6mREREREpCi6etdlWRNlsb5e32YaVNLavHrLZEv42u7Jcc6NsUJSpRzE3uLvDuXP5v7m7Z6vMnj178u2339pniJ05cybdu3fHarVy7tw5XnnlFerWrUvp0qUpUaIE27dvz3GP4s8//8xdd91FxYoV8fT05LHHHuP48eOc/7cXNL1HMTPbt28nOTn5uq9nlbOzMw0bNszQlpiYyFNPPUXNmjUpVaoUJUuW5Ny5c/brjImJoVKlStSqVSvTczZv3px69eoxffp0AL766iuqVKlC69atb6lWERERkULJpQS0GQgvxkCzJ8HqCLuWwqRWsLAvnI43u0LJIQXF3GCxgIdH/m8WS7bK7NSpEzabjR9++IEDBw7w22+/0bNnTwBeeeUVFixYwKhRo/jtt9+IiYmhQYMGOZopdt++fdx33300bNiQb7/9lo0bNzJx4kQA+/nc3Nyue/yNXgPsE9LYrhgHf+nSpUzPY7nqe9S7d29iYmL48MMPWbNmDTExMZQtWzZLdaV78skn7ffNfvHFF0RERFzzPiIiIiLFSgkfYyhqv3UQ2BWwQcxMGN8Elg2F1Gt/V5OCTUGxGHF1daVbt27MnDmT2bNnU7t2bRo3bgwYE8v06dOH+++/nwYNGuDn52efGCa7Nm7cSFpaGmPGjOG2226jVq1aHDp0KMM+DRs2zLB+5pVq1qyJm5vbdV8vV64cAIcPH7a3xcTEZKm21atX88ILL3DvvfdSr149XFxcOHbsWIa64uPj2blz53XP8eijj7J//34++ugjtm3bRu/evbP03iIiIiJFXtnq8PB0eHI5VLkdLl+Eg5uMnkYpVPQTK2Z69uzJfffdx9atW+1rT4IRzubPn0+nTp2wWCwMGTLkmhlSs6pGjRpcunSJ8ePH06lTJ1avXs3kyZMz7DNo0CAaNGhA3759efbZZ3F2dmbFihU89NBDeHt78/rrr/Paa6/h7OxMq1atOHr0KFu3buWJJ56gRo0a+Pv7M3z4cN555x127tzJmDFjslRbzZo1mTFjBk2bNuXMmTO8+uqrGXoR77zzTlq3bs0DDzzA2LFjqVGjBjt27MBisdChQwcAypQpQ7du3Xj11Ve5++67qVSpUo6+TyIiIiJFVqUm0GcR7PoJPMtneyScmE89isVMu3bt8PLyIjY2lkceecTePnbsWMqUKUPLli3p1KkTYWFh9t7G7AoKCmLs2LG899571K9fn5kzZxIVFZVhn1q1avHTTz/x559/0rx5c1q0aMF3332Ho6Pxt4shQ4YwYMAAhg4dSt26dQkPD+fIkSMAODk5MXv2bHbs2EHDhg157733ePvtt7NU2+eff87Jkydp3Lgxjz32GC+88AI+Pj4Z9vn2229p1qwZPXr0IDAwkNdee80+G2u6J554gpSUFB5//PEcfY9EREREijyLBWqFQfmGN99XChyL7cobvQSA+Ph4/P39OXDgwDW9RRcvXmTv3r1UrVoVV1dXkyoUs82YMYOXX36ZQ4cO4ezsfMN99ZkRERERKbxulA2KMg09FcmG8+fPc/jwYd59912eeeaZm4ZEEREREZHCSENPJdtmzpxJiRIlMt3q1atndnl5avTo0dSpUwc/Pz8GDRpkdjkiIiIiInlCPYqSbZ07dyYkJCTT15ycnPK5mvw1fPhwhg8fbnYZIiIiIiJ5SkFRss3T0xNPT0+zyxARERERkTyioac5pDmAJKv0WRERERGRwkZBMZvSh1aeP3/e5EqksEj/rBT1YbkiIiIiUnRo6Gk2OTg4ULp0afuafu7u7li0gKhkwmazcf78eY4cOULp0qVxcHAwuyQRERERkSxRUMwBPz8/AHtYFLmR0qVL2z8zIiIiIiKFgYJiDlgsFsqXL4+Pjw+XLl0yuxwpwJycnNSTKCIiIiKFjoLiLXBwcFAIEBERERGRIkeT2YiIiIiIiOSyiRMnEhAQgKurKyEhIaxbt+6G+3/zzTfUqVMHV1dXGjRowOLFi/Op0swpKIqIiIiIiOSiuXPnEhkZybBhw9i0aRNBQUGEhYVdd46TNWvW0KNHD5544gk2b95M165d6dq1K3///Xc+V/4fi02LvF0jPj4ef39/Dhw4QKVKlcwuR0RERERETJKTbBASEkKzZs2YMGECAGlpafj7+9O/f38GDhx4zf7h4eEkJSWxaNEie9ttt91GcHAwkydPzp0LySbdo5iJtLQ0AA4fPmxyJSIiIiIiYqb0THD69GlKlixpb3dxccHFxeWa/VNSUti4cSODBg2yt1mtVkJDQ1m7dm2m77F27VoiIyMztIWFhbFw4cJcuIKcUVDMRGJiIgDNmzc3uRIRERERESkI6tevn+H5sGHDGD58+DX7HTt2jNTUVHx9fTO0+/r6smPHjkzPnZCQkOn+CQkJt1b0LVBQzESjRo1Yt24dvr6+WK3m3sZ59uxZAgMD2bZtG56enqbWIoWDPjOSXfrMSHbpMyPZpc+MZFdB+sykpaURFxdHYGAgjo7/xafMehOLEgXFTDg6OtKsWTOzywDgzJkzAFSsWDFDV7fI9egzI9mlz4xklz4zkl36zEh2FbTPTOXKlbO8r7e3Nw4ODvZRiukSExPx8/PL9Bg/P79s7Z8fNOupiIiIiIhILnF2dqZJkyZER0fb29LS0oiOjqZFixaZHtOiRYsM+wMsW7bsuvvnB/UoioiIiIiI5KLIyEh69+5N06ZNad68OePGjSMpKYmIiAgAevXqRcWKFYmKigLgxRdf5M4772TMmDF07NiROXPmsGHDBj799FPTrkFBsYBzcXFh2LBhRX4MtOQefWYku/SZkezSZ0ayS58Zya7C/pkJDw/n6NGjDB06lISEBIKDg1myZIl9wpq4uLgMc6G0bNmSWbNm8eabb/LGG29Qs2ZNFi5ceM0EOvlJ6yiKiIiIiIhIBrpHUURERERERDJQUBQREREREZEMFBRFREREREQkAwVFERERERERyUBBsQCYOHEiAQEBuLq6EhISwrp16264/zfffEOdOnVwdXWlQYMGLF68OJ8qlYIiO5+ZKVOmcMcdd1CmTBnKlClDaGjoTT9jUvRk9/8z6ebMmYPFYqFr1655W6AUONn9zJw6dYp+/fpRvnx5XFxcqFWrlv59Kmay+5kZN24ctWvXxs3NDX9/f15++WUuXryYT9WKmX799Vc6depEhQoVsFgsLFy48KbHrFy5ksaNG+Pi4kKNGjWYNm1antdZ3Ckommzu3LlERkYybNgwNm3aRFBQEGFhYRw5ciTT/desWUOPHj144okn2Lx5M127dqVr1678/fff+Vy5mCW7n5mVK1fSo0cPVqxYwdq1a/H39+fuu+/m4MGD+Vy5mCW7n5l0+/bt45VXXuGOO+7Ip0qloMjuZyYlJYX27duzb98+5s2bR2xsLFOmTKFixYr5XLmYJbufmVmzZjFw4ECGDRvG9u3b+fzzz5k7dy5vvPFGPlcuZkhKSiIoKIiJEydmaf+9e/fSsWNH2rZtS0xMDC+99BJPPvkkS5cuzeNKizmbmKp58+a2fv362Z+npqbaKlSoYIuKisp0/4cfftjWsWPHDG0hISG2Z555Jk/rlIIju5+Zq12+fNnm6elpmz59el6VKAVMTj4zly9ftrVs2dL22Wef2Xr37m3r0qVLPlQqBUV2PzOTJk2yVatWzZaSkpJfJUoBk93PTL9+/Wzt2rXL0BYZGWlr1apVntYpBQ9gW7BgwQ33ee2112z16tXL0BYeHm4LCwvLw8pEPYomSklJYePGjYSGhtrbrFYroaGhrF27NtNj1q5dm2F/gLCwsOvuL0VLTj4zVzt//jyXLl3Cy8srr8qUAiSnn5kRI0bg4+PDE088kR9lSgGSk8/M999/T4sWLejXrx++vr7Ur1///9u505Co/j2O459JnXHJaBF1iha0zaKFdiuI6kELFEVRkQwahIgVPSmKFsawQiLqQZRRtDwokor6I2W2WD3IiKLUhNQwaYGaSgoy25vffdBt/vf8tXuvQ84xfb/ggPM7v6OfH3wdznfOnKPt27fr+/fvoYoNGwVTMxMnTtTdu3cDX0+tq6tTUVGRZs+eHZLM+LNw/muPcLsDdGT19fX6/v27EhISLOMJCQmqrq5u9hifz9fsfJ/P12o50XYEUzP/tG7dOvXs2bPJGy7ap2Bq5saNGzp06JDKy8tDkBBtTTA1U1dXp6tXryotLU1FRUWqra1Vdna2vn79Kq/XG4rYsFEwNbN06VLV19dr8uTJMsbo27dvysrK4qunaNavzn/fvXunjx8/KioqyqZk7RtXFIEOJC8vTwUFBTp79qwiIyPtjoM2qKGhQR6PRwcPHlRcXJzdcfCH8Pv9io+P14EDBzR69GgtXrxYGzdu1P79++2Ohjbq+vXr2r59u/bt26d79+7pzJkzOn/+vHJzc+2OBuDfuKJoo7i4OIWFhenly5eW8ZcvXyoxMbHZYxITE1s0H+1LMDXz086dO5WXl6crV65o+PDhrRkTbUhLa+bRo0d6/Pix5syZExjz+/2SpPDwcNXU1Cg5Obl1Q8NWwbzPuN1uRUREKCwsLDCWkpIin8+nL1++yOl0tmpm2CuYmtm8ebM8Ho+WL18uSRo2bJgaGxuVmZmpjRs3qlMnrmXgb786/+3SpQtXE1sR/4U2cjqdGj16tEpKSgJjfr9fJSUlSk1NbfaY1NRUy3xJunz58i/no30JpmYkaceOHcrNzVVxcbHGjBkTiqhoI1paM4MHD1ZlZaXKy8sD29y5cwNPmuvdu3co48MGwbzPTJo0SbW1tYEPFSTp4cOHcrvdNIkdQDA18+HDhybN4M8PGowxrRcWfyTOf21i99N0OrqCggLjcrnM0aNHzYMHD0xmZqbp2rWr8fl8xhhjPB6PWb9+fWB+aWmpCQ8PNzt37jRVVVXG6/WaiIgIU1lZadcSEGItrZm8vDzjdDrN6dOnzYsXLwJbQ0ODXUtAiLW0Zv6Jp552PC2tmadPn5rY2FizcuVKU1NTY86dO2fi4+PN1q1b7VoCQqylNeP1ek1sbKw5ceKEqaurM5cuXTLJyclm0aJFdi0BIdTQ0GDKyspMWVmZkWR27dplysrKzJMnT4wxxqxfv954PJ7A/Lq6OhMdHW3Wrl1rqqqqzN69e01YWJgpLi62awkdAo1iG7Bnzx7Tp08f43Q6zbhx48ytW7cC+6ZMmWLS09Mt80+ePGkGDhxonE6nGTp0qDl//nyIE8NuLamZvn37GklNNq/XG/rgsE1L32f+E41ix9TSmrl586YZP368cblcJikpyWzbts18+/YtxKlhp5bUzNevX01OTo5JTk42kZGRpnfv3iY7O9u8ffs29MERcteuXWv23ORnjaSnp5spU6Y0OWbkyJHG6XSapKQkc+TIkZDn7mgcxnB9HwAAAADwN+5RBAAAAABY0CgCAAAAACxoFAEAAAAAFjSKAAAAAAALGkUAAAAAgAWNIgAAAADAgkYRAAAAAGBBowgAAAAAsKBRBADgN3M4HPrrr7/sjgEAQNBoFAEA7UpGRoYcDkeTbebMmXZHAwDgjxFudwAAAH63mTNn6siRI5Yxl8tlUxoAAP48XFEEALQ7LpdLiYmJlq1bt26SfnwtND8/X7NmzVJUVJSSkpJ0+vRpy/GVlZWaNm2aoqKi1KNHD2VmZur9+/eWOYcPH9bQoUPlcrnkdru1cuVKy/76+nrNnz9f0dHRGjBggAoLC1t30QAA/EY0igCADmfz5s1asGCBKioqlJaWpiVLlqiqqkqS1NjYqBkzZqhbt266c+eOTp06pStXrlgawfz8fK1YsUKZmZmqrKxUYWGh+vfvb/kbW7Zs0aJFi3T//n3Nnj1baWlpevPmTUjXCQBAsBzGGGN3CAAAfpeMjAwdO3ZMkZGRlvENGzZow4YNcjgcysrKUn5+fmDfhAkTNGrUKO3bt08HDx7UunXr9OzZM8XExEiSioqKNGfOHD1//lwJCQnq1auXli1bpq1btzabweFwaNOmTcrNzZX0o/ns3LmzLly4wL2SAIA/AvcoAgDanalTp1oaQUnq3r174OfU1FTLvtTUVJWXl0uSqqqqNGLEiECTKEmTJk2S3+9XTU2NHA6Hnj9/runTp//XDMOHDw/8HBMToy5duujVq1fBLgkAgJCiUQQAtDsxMTFNvgr6u0RFRf1f8yIiIiyvHQ6H/H5/a0QCAOC34x5FAECHc+vWrSavU1JSJEkpKSmqqKhQY2NjYH9paak6deqkQYMGKTY2Vv369VNJSUlIMwMAEEpcUQQAtDufP3+Wz+ezjIWHhysuLk6SdOrUKY0ZM0aTJ0/W8ePHdfv2bR06dEiSlJaWJq/Xq/T0dOXk5Oj169datWqVPB6PEhISJEk5OTnKyspSfHy8Zs2apYaGBpWWlmrVqlWhXSgAAK2ERhEA0O4UFxfL7XZbxgYNGqTq6mpJP55IWlBQoOzsbLndbp04cUJDhgyRJEVHR+vixYtavXq1xo4dq+joaC1YsEC7du0K/K709HR9+vRJu3fv1po1axQXF6eFCxeGboEAALQynnoKAOhQHA6Hzp49q3nz5tkdBQCANot7FAEAAAAAFjSKAAAAAAAL7lEEAHQo3HEBAMD/xhVFAAAAAIAFjSIAAAAAwIJGEQAAAABgQaMIAAAAALCgUQQAAAAAWNAoAgAAAAAsaBQBAAAAABY0igAAAAAAi38BVjm9fqfcTD0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get loss, val_loss, and the computed metric from history\n",
    "loss = [x['loss'] for x in history if 'loss' in x]\n",
    "val_loss = [x['eval_loss'] for x in history if 'eval_loss' in x]\n",
    "\n",
    "# Get accuracy value \n",
    "metric = [x['eval_accuracy'] for x in history if 'eval_accuracy' in x]\n",
    "\n",
    "epochs_loss = [x['epoch'] for x in history if 'loss' in x]\n",
    "epochs_eval = [x['epoch'] for x in history if 'eval_loss' in x]\n",
    "\n",
    "# Create a figure with two y-axes\n",
    "fig, ax1 = plt.subplots(figsize=(10, 5))\n",
    "ax2 = ax1.twinx()\n",
    "\n",
    "# Plot loss and val_loss on the first y-axis\n",
    "# For the loss we plot a horizontal line because we have just one loss value (after the first epoch)\n",
    "# Exchange the two lines below if you trained multiple epochs\n",
    "line1 = ax1.plot([0]+epochs_loss, loss*2, label='train_loss')\n",
    "#line1 = ax1.plot(epochs_loss, loss, label='train_loss')\n",
    "\n",
    "line2 = ax1.plot(epochs_eval, val_loss, label='val_loss')\n",
    "ax1.set_xlabel('Epoch')\n",
    "ax1.set_ylabel('Loss')\n",
    "\n",
    "# Plot the computed metric on the second y-axis\n",
    "line3 = ax2.plot(epochs_eval, metric, color='red', label='val_accuracy')\n",
    "ax2.set_ylabel('Accuracy')\n",
    "ax2.set_ylim([0, 1])\n",
    "\n",
    "# Combine the lines from both y-axes and create a single legend\n",
    "lines = line1 + line2 + line3\n",
    "labels = [line.get_label() for line in lines]\n",
    "ax1.legend(lines, labels, loc='lower left')\n",
    "\n",
    "# Show the plot\n",
    "plt.title(\"Training History\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a4a53e",
   "metadata": {},
   "source": [
    "# Save and Load the finetuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6ade4a9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_model(model,filepath):\n",
    "# Saves all parameters that were changed during finetuning\n",
    "\n",
    "    # Create a dictionary to hold the non-frozen parameters\n",
    "    non_frozen_params = {}\n",
    "\n",
    "    # Iterate through all the model parameters\n",
    "    for param_name, param in model.named_parameters():\n",
    "        # If the parameter has requires_grad=True, add it to the dictionary\n",
    "        if param.requires_grad:\n",
    "            non_frozen_params[param_name] = param\n",
    "\n",
    "    # Save only the finetuned parameters \n",
    "    torch.save(non_frozen_params, filepath)\n",
    "\n",
    "    \n",
    "def load_model(filepath, num_labels=1):\n",
    "# Creates a new PT5 model and loads the finetuned weights from a file\n",
    "\n",
    "    # load a new model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels)\n",
    "    \n",
    "    # Load the non-frozen parameters from the saved file\n",
    "    non_frozen_params = torch.load(filepath)\n",
    "\n",
    "    # Assign the non-frozen parameters to the corresponding parameters of the model\n",
    "    for param_name, param in model.named_parameters():\n",
    "        if param_name in non_frozen_params:\n",
    "            param.data = non_frozen_params[param_name].data\n",
    "\n",
    "    return tokenizer, model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5ba621",
   "metadata": {},
   "source": [
    "This saves only the finetuned weights to a .pth file\n",
    "\n",
    "It is a 10 MB File, while the entire model would be around 4.8 GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d31dc7e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_model(model,\"./PT5_secstr_finetuned.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8e96aa",
   "metadata": {},
   "source": [
    "To load the weights again, we initialize a new PT5 model from the pretrained checkpoint and load the LoRA weights afterwards\n",
    "\n",
    "You need to specifiy the correct num_labels here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "edbd69f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/pytorch_model.bin\n",
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.embed_tokens.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.final_layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'lm_head.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All the weights of T5EncoderModel were initialized from the model checkpoint at Rostlab/prot_t5_xl_uniref50.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use T5EncoderModel for predictions without further training.\n",
      "loading file spiece.model from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/spiece.model\n",
      "loading file added_tokens.json from cache at None\n",
      "loading file special_tokens_map.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/special_tokens_map.json\n",
      "loading file tokenizer_config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/tokenizer_config.json\n",
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"_name_or_path\": \"Rostlab/prot_t5_xl_uniref50\",\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1208144899\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 2510851\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model_reload = load_model(\"./PT5_secstr_finetuned.pth\", num_labels=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5289780",
   "metadata": {},
   "source": [
    "To check if the original and the reloaded models are identical we can compare weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e152714e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Models have identical weights\n"
     ]
    }
   ],
   "source": [
    "# Put both models to the same device\n",
    "model=model.to(\"cpu\")\n",
    "model_reload=model_reload.to(\"cpu\")\n",
    "\n",
    "# Iterate through the parameters of the two models and compare the data\n",
    "for param1, param2 in zip(model.parameters(), model_reload.parameters()):\n",
    "    if not torch.equal(param1.data, param2.data):\n",
    "        print(\"Models have different weights\")\n",
    "        break\n",
    "else:\n",
    "    print(\"Models have identical weights\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c478158",
   "metadata": {},
   "source": [
    "# Make predictions on a test set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08533e20",
   "metadata": {},
   "source": [
    "This time we take the test data we prepared before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1151f4de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M T P A V T T Y K L V I N G K T L K G E T T T ...</td>\n",
       "      <td>[-100, -100, -100, -100, 0, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M N D Q E K I D K F T H S Y I N D D F G L T I ...</td>\n",
       "      <td>[-100, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 1, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>G P G F M R D S G S K A S S D S Q D A N Q C C ...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>S N A L S R N E V L L N G D I N F K E V R C V ...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M A K G K S E V V E Q N H T L I L G W S D K L ...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -10...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence  \\\n",
       "0  M T P A V T T Y K L V I N G K T L K G E T T T ...   \n",
       "1  M N D Q E K I D K F T H S Y I N D D F G L T I ...   \n",
       "2  G P G F M R D S G S K A S S D S Q D A N Q C C ...   \n",
       "3  S N A L S R N E V L L N G D I N F K E V R C V ...   \n",
       "4  M A K G K S E V V E Q N H T L I L G W S D K L ...   \n",
       "\n",
       "                                               label  \n",
       "0  [-100, -100, -100, -100, 0, 1, 1, 1, 1, 1, 1, ...  \n",
       "1  [-100, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 1, ...  \n",
       "2  [-100, -100, -100, -100, -100, -100, -100, -10...  \n",
       "3  [-100, -100, -100, -100, -100, -100, -100, 0, ...  \n",
       "4  [-100, -100, -100, -100, -100, -100, -100, -10...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop unneeded columns (remember, mask was already included as -100 values to label)\n",
    "my_test=my_test[[\"sequence\",\"label\"]]\n",
    "\n",
    "# Preprocess sequences\n",
    "my_test[\"sequence\"]=my_test[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "my_test['sequence']=my_test.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "my_test.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9249f897",
   "metadata": {},
   "source": [
    "Then we create predictions on our test data using the model we trained before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1df2299",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 23/23 [01:34<00:00,  4.13s/it]\n"
     ]
    }
   ],
   "source": [
    "# Set the device to use\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "model.to(device)\n",
    "\n",
    "# Create Dataset\n",
    "test_set=create_dataset(tokenizer,list(my_test['sequence']),list(my_test['label']))\n",
    "# Make compatible with torch DataLoader\n",
    "test_set = test_set.with_format(\"torch\", device=device)\n",
    "\n",
    "# For token classification we need a data collator here to pad correctly\n",
    "data_collator = DataCollatorForTokenClassification(tokenizer) \n",
    "\n",
    "# Create a dataloader for the test dataset\n",
    "test_dataloader = DataLoader(test_set, batch_size=16, shuffle = False, collate_fn = data_collator)\n",
    "\n",
    "# Put the model in evaluation mode\n",
    "model.eval()\n",
    "\n",
    "# Make predictions on the test dataset\n",
    "predictions = []\n",
    "# We need to collect the batch[\"labels\"] as well, this allows us to filter out all positions with a -100 afterwards\n",
    "padded_labels = []\n",
    "\n",
    "with torch.no_grad():\n",
    "    for batch in tqdm(test_dataloader):\n",
    "        input_ids = batch['input_ids'].to(device)\n",
    "        attention_mask = batch['attention_mask'].to(device)\n",
    "        # Padded labels from the data collator\n",
    "        padded_labels += batch['labels'].tolist()\n",
    "        # Add batch results(logits) to predictions, we take the argmax here to get the predicted class\n",
    "        predictions += model(input_ids, attention_mask=attention_mask).logits.argmax(dim=-1).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18401de",
   "metadata": {},
   "source": [
    "Finally, we compute our desired performance metric for the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "000028da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.8460932598219173\n"
     ]
    }
   ],
   "source": [
    "# to make it easier we flatten both the label and prediction lists\n",
    "def flatten(l):\n",
    "    return [item for sublist in l for item in sublist]\n",
    "\n",
    "# flatten and convert to np array for easy slicing in the next step\n",
    "predictions = np.array(flatten(predictions))\n",
    "padded_labels = np.array(flatten(padded_labels))\n",
    "\n",
    "# Filter out all invalid (label = -100) values\n",
    "predictions = predictions[padded_labels!=-100]\n",
    "padded_labels = padded_labels[padded_labels!=-100]\n",
    "\n",
    "# Calculate classification Accuracy\n",
    "print(\"Accuracy: \", accuracy_score(padded_labels, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee716709",
   "metadata": {},
   "source": [
    "Great, 84.6% Accuracy is a decent test performance for the \"new_pisces\" dataset (see results in [Table 7](https://ieeexplore.ieee.org/ielx7/34/9893033/9477085/supp1-3095381.pdf?arnumber=9477085) \"NEW364\" )\n",
    "\n",
    "Further hyperparameter optimization and using a CNN prediction head will further increase performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "013dd2a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [
    {
     "file_id": "https://github.com/huggingface/notebooks/blob/main/examples/protein_language_modeling-tf.ipynb",
     "timestamp": 1670229986129
    }
   ]
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "023504ef79df47cd9f5672b3537d781e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f3f0b29c16df48f798dbdabc99207f2c",
       "IPY_MODEL_fb9ce840b4644d6eab03736688e57e23",
       "IPY_MODEL_dcb559ab088446f0a4cc9ae4720f8c29"
      ],
      "layout": "IPY_MODEL_cf11d66697d84676b9d8af11a2edd064"
     }
    },
    "07e6f8ff49d74ced96b87afaf99f52c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a7bab101d504c7cbe0c0e21222e4010": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6e739bd554404481b4cfac9cd1710224",
      "max": 95,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b66c522257d04820932c9c7014392136",
      "value": 95
     }
    },
    "0c9bd61e2e904fdaa4f578c19c0f6ea9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "14ba9ccca4004a1e944767648e2b1253": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d04e61e6abfe42b9afab583bcab66b92",
      "placeholder": "​",
      "style": "IPY_MODEL_279e487c47954c9abc0a57daab09f66a",
      "value": "Downloading: 100%"
     }
    },
    "14eaa8780fd543b38d5ea5da8e002b1e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9a28a789e7814ee283b8ba922ecce252",
      "placeholder": "​",
      "style": "IPY_MODEL_3149e4afc57146ceac1fdd9a1902ccbc",
      "value": "Downloading: 100%"
     }
    },
    "15e24c603abb456984509d77813a0cf7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9ef0944bc4334b32a1a5e136b3fce9e6",
      "placeholder": "​",
      "style": "IPY_MODEL_c6407f5a50b34a0d9bfbd916ffb70d17",
      "value": " 134M/134M [00:01&lt;00:00, 79.0MB/s]"
     }
    },
    "1bdf17da40fb4d8687456e5379af5da9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1f5719ff48504d91968e1953107b060f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "279e487c47954c9abc0a57daab09f66a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2f1f8e5d1f5343978ccf1cc5dff59718": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "305e2c8af6194f4486b505ff25be86d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_611417b8118744c28ec2619c45b2a6ec",
      "max": 125,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cd89abbac71740c5a013481dd9129f5b",
      "value": 125
     }
    },
    "3149e4afc57146ceac1fdd9a1902ccbc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "334e77a8b6284dfb886fefadea6998ca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "37dce533ad7648d0a09b3ef825c5aa31": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d579e0c846b4e1bbc360d0ff2f2eccf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4db0117d9ac0453886964613627decc0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "51b29673275343998567b43347eb591b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "52a93c8ac4454ed1bf1a0d4711074bc2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d154e8e8a650447ab06425706d908f4e",
       "IPY_MODEL_0a7bab101d504c7cbe0c0e21222e4010",
       "IPY_MODEL_c8b35d958ecc4f3683b86278b979f300"
      ],
      "layout": "IPY_MODEL_4db0117d9ac0453886964613627decc0"
     }
    },
    "611417b8118744c28ec2619c45b2a6ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6ae302430d7746edbc04b6786bdc53f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14ba9ccca4004a1e944767648e2b1253",
       "IPY_MODEL_e499b443a3964dc2bca05692abde2dda",
       "IPY_MODEL_15e24c603abb456984509d77813a0cf7"
      ],
      "layout": "IPY_MODEL_1f5719ff48504d91968e1953107b060f"
     }
    },
    "6c3f28dcd36d434fa8e5fe38b1d34e3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8d4673cb688f4dacb1e96a0d80815667",
       "IPY_MODEL_305e2c8af6194f4486b505ff25be86d7",
       "IPY_MODEL_a6f664af44464825a9c8b73215b59318"
      ],
      "layout": "IPY_MODEL_2f1f8e5d1f5343978ccf1cc5dff59718"
     }
    },
    "6e739bd554404481b4cfac9cd1710224": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "728fbf87d2ff4bed925f67c9cf0e36b7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a00dda470ef4f0893676c794e006a8f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8921dca0901741c5ab7723ca88803543": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14eaa8780fd543b38d5ea5da8e002b1e",
       "IPY_MODEL_abee545f24074d33a2a171f140d84ca4",
       "IPY_MODEL_d2d78bbb74664973b1cde561152e9aad"
      ],
      "layout": "IPY_MODEL_fefc4de84c7847bf88635bfcad3a5f9e"
     }
    },
    "8d4673cb688f4dacb1e96a0d80815667": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_728fbf87d2ff4bed925f67c9cf0e36b7",
      "placeholder": "​",
      "style": "IPY_MODEL_97335231c9ab4668bc8cdaee57be202f",
      "value": "Downloading: 100%"
     }
    },
    "97335231c9ab4668bc8cdaee57be202f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9a28a789e7814ee283b8ba922ecce252": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9ef0944bc4334b32a1a5e136b3fce9e6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6f664af44464825a9c8b73215b59318": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fc1cf1de39dc45d69551ca2ca401a447",
      "placeholder": "​",
      "style": "IPY_MODEL_1bdf17da40fb4d8687456e5379af5da9",
      "value": " 125/125 [00:00&lt;00:00, 1.34kB/s]"
     }
    },
    "a7e27116c7b04f5e8980ea29c59bd9ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a840d7d2dc0042c1955f064ef41e051d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "abee545f24074d33a2a171f140d84ca4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_07e6f8ff49d74ced96b87afaf99f52c0",
      "max": 778,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ba9fc00d9d144278bcf944a9328d5d5f",
      "value": 778
     }
    },
    "b66c522257d04820932c9c7014392136": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ba9fc00d9d144278bcf944a9328d5d5f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c555968966fb402e9ed39c024476f34b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c6407f5a50b34a0d9bfbd916ffb70d17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c8b35d958ecc4f3683b86278b979f300": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_37dce533ad7648d0a09b3ef825c5aa31",
      "placeholder": "​",
      "style": "IPY_MODEL_e16a79b61af94733b44306e7b5444d29",
      "value": " 95.0/95.0 [00:00&lt;00:00, 3.19kB/s]"
     }
    },
    "cd89abbac71740c5a013481dd9129f5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cf11d66697d84676b9d8af11a2edd064": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d015260b083c40399f87d1a30be95f4f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d04e61e6abfe42b9afab583bcab66b92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d154e8e8a650447ab06425706d908f4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ef762a4251b6405f8065c3e4ce880688",
      "placeholder": "​",
      "style": "IPY_MODEL_4d579e0c846b4e1bbc360d0ff2f2eccf",
      "value": "Downloading: 100%"
     }
    },
    "d2d78bbb74664973b1cde561152e9aad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_51b29673275343998567b43347eb591b",
      "placeholder": "​",
      "style": "IPY_MODEL_334e77a8b6284dfb886fefadea6998ca",
      "value": " 778/778 [00:00&lt;00:00, 26.3kB/s]"
     }
    },
    "dcb559ab088446f0a4cc9ae4720f8c29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_eb8017bf12034a0297f17f0f3f5f1874",
      "placeholder": "​",
      "style": "IPY_MODEL_a840d7d2dc0042c1955f064ef41e051d",
      "value": " 93.0/93.0 [00:00&lt;00:00, 3.23kB/s]"
     }
    },
    "e16a79b61af94733b44306e7b5444d29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e499b443a3964dc2bca05692abde2dda": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ebb2f370c71a470895096be1ab1eadbf",
      "max": 134360208,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a7e27116c7b04f5e8980ea29c59bd9ce",
      "value": 134360208
     }
    },
    "eb8017bf12034a0297f17f0f3f5f1874": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ebb2f370c71a470895096be1ab1eadbf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef762a4251b6405f8065c3e4ce880688": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f3f0b29c16df48f798dbdabc99207f2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7a00dda470ef4f0893676c794e006a8f",
      "placeholder": "​",
      "style": "IPY_MODEL_0c9bd61e2e904fdaa4f578c19c0f6ea9",
      "value": "Downloading: 100%"
     }
    },
    "fb9ce840b4644d6eab03736688e57e23": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d015260b083c40399f87d1a30be95f4f",
      "max": 93,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c555968966fb402e9ed39c024476f34b",
      "value": 93
     }
    },
    "fc1cf1de39dc45d69551ca2ca401a447": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fefc4de84c7847bf88635bfcad3a5f9e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
